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Abstract. We study a new Selberg-type integral with n + m indeterminates, which turns out to be 
related to the deformed Calogero-Sutherland systems. We show that the integral satisfies a holonomic 
system of n + m non-symmetric linear partial differential equations. We also prove that a particular 
hypergeometric function denned in terms of super Jack polynomials is the unique solution of the system. 
Some properties such as duality relations, integral formulas, Pfaff-Eulcr and Kummcr transformations 
are also established. As a direct application, we evaluate the expectation value of ratios of characteristic 
polynomials in the classical /3-ensembles of Random Matrix Theory. 
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1. Introduction 

1.1. Kaneko's integral and symmetric polynomials. In the early 1990s, Kaneko [21] thoroughly 
studied the following generalization of the Selberg integral: 

„ N n 

K N (X 1 ,X 2 ,X;t) = D Xl>XatX (x)J T {x 3 <- t k )» d N x (1.1) 

•WP j=ik=i 

where t denotes the set of variables t\, . . . ,t n , the parameter \i is either equal to 1 or —A, and 

TV 

D XuX2 ^(x)=l[x^(l~x l ) x i J] \x,-x k \ 2X . (1-2) 

i=l l<j<k<N 

Note that a convenient way to guarantee the convergence of the integral is to set 3J(A) > 0, 3f(Ai) > — 1, 
3?(A2) > — 1, and for the case with \i = — A, i; 6 C \ [0, 1]. In the absence of variables t, which corresponds 
to the case n = 0, the value of the integral is given by Selberg's celebrated result [33] 

q i \ \ \\ ^r 1 r(i + a + jA)r(i + x 1+ , ? A)r(i + a 2 + jX) 

5Ar(Al ' A2 ' A) 11 r(i + A)r(2 + A 1 + A 2 + (Jv + j-i)A) ■ (L3) 

See |15j for a beautiful discussion on the above formula and its various applications. 

Kaneko showed in particular that the integral K N (Xi, A 2 , A; t) satisfies the following holonomic system 
of partial differential equations: 

x d 2 F ( n-l\dF ( , , n-l\ dF 

u ( 1 - t ^ + { c -—)wr{ a+b+1 -—) u ^- abF 

1 1 / BP r)F\ 

^E^ '4l-^-ti(l-^ =0, i = l,...,n, (1.4) 

where a = A, a = -N, b = (Ai + A 2 + n + 1)/A + N - 1, c = (Ai +n)/X if fx = 1, while a = 1/A, a = XN, 
b = — (Ai + X-2 + 1) + A(n — N + 1), c = — Ai + nA if yu = —A. He further proved that the above system 
possesses a unique solution F(t\, . . . ,t n ) subject to the following conditions: 

(1) F(ti, . . . , t n ) is symmetric of ti, . . . ,t n ; 

(2) . .,t n ) is analytic at (0, ... ,0) and such that F(0, ... ,0) = 1. 

This allowed him to conclude for instance that the integral Km given in Eq. (|1.1|) with \i = 1 is equal to 
the Selberg constant ^^(Ai +?^,A2,A) multiplied by a Gaussian hypergeometric function in n variables 
2-P\ (a, 6; c; ti, . . . , t n ). As will be explained in Section[31 the precise definition of the latter function requires 
some theory on symmetric Jack polynomials, a subject that was developed in the late 1980s, principally 
by Kadell, Macdonald, and Stanley [5U1HH1I11] . Remarkably, the theorem on the existence and uniqueness 
of the solution for the system (|1.4|) was independently and almost simultaneously obtained by Yan [46j . 
Although not concerned with the full generalized Selberg integral (jl.ip . Yan obtained a series of results 
on hypergeometric functions also defined in terms of Jack polynomial together with some of their integral 
representations . 

One of the main inspirations behind Kaneko's work was an article by Aomoto on a one-variable gener- 
alization of the Selberg integral and its relation to the Jacobi polynomials pQ. In our notation, Aomoto's 
integral corresponds to ifjv(Ai, A2, A; ti) (i.e., case n = 1). Another crucial motivation for studying the 
integral (jl.ip was Muirhead's holonomic system of equations |35j whose unique solution was given as a 
hypergeometric function 2F1 (a, b; c; T) whose argument T is a real symmetric n x n matrix. For special 
values of parameters a, 6, c, the latter function can be represented as an integral over some symmetric 
N x N matrix, which in turn is closely related to the integral (jl.ip with n general but with A = 1/2 [36] . 

All the previous results converged in the seminal article by Baker and Forrester [2]. There, the authors 
used the hypergeometric functions of Kaneko and Yan for studying the Hermite, Laguerre and Jacobi 
polynomials in many variables as independently defined by Lassallc and Macdonald, both using Jack 
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polynomial theory. They also showed that the classical polynomials in many variables as well as their 
associated hypergeometric functions are all related to the famous quantum many systems of Calogero and 
Sutherland and applied their results for evaluating the ground state density of these systems, which, as is 
well known, is equivalent to the eigenvalue density of classical ensembles of random matrices. 

1.2. Relations with Random Matrix Theory. The integrand (|1.2[) is indeed known in Random Matrix 
Theory |14U16[|32] as the non-normalized eigenvalue probability density function in the Jacobi /3-Ensemble, 
where j3 = 2A is any positive real. The latter ensemble is a generalization of the three classical Jacobi 
Ensembles which can been defined in terms of rectangular matrices with real (j3 = 1), complex (/? = 2) 
or real quaternions (J3 = 2) elements jT4j Chapter 3]. Thanks to the parameters Ai and A2, the Jacobi 
Ensembles can also be seen as generalizing the somewhat more common Circular, Gaussian and Chiral 
Ensembles of random matrices, whose respective eigenvalue probability densities are proportional to 



n 



| B j — e ~ |<- Xj <e[0,1] Circular (1.5) 

l<j<fe<Af 

N 

~[ e - f3x2 > /2 \xj~Xkf xj G K Gaussian (Hermite) (1.6) 

i=l l<j<k<N 
N 

~[x$ 1 e~ l3xi \xj~x k f i,ei + Chiral (Laguerre). (1.7) 

i=l l<j<k<N 



When (3 is any real positive number, the function ()1.2|) is proportional to the eigenvalue density coming 
from a real symmetric matrix whose non-zero elements lay on the three main diagonals and are randomly 
drawn from some specific distributions. This was first obtained in the article |25j by extending the work 
of Dumitriu and Edclman [13] on Laguerre and Hermite /3-Enscmbles as models of tri-diagonal random 
matrices. 

The properties and applications of /3-Enscmblcs have been studied by many other authors. For instance, 
it was established that in the limit where the size N of a random matrix /3-Enscmbles goes to infinity, 
the eigenvalues of the random matrix become statistically distributed as the eigenvalues of some one- 
dimensional stochastic differential operators [35}[3S]. In high energy physics, the /3-Ensembles recently 
appeared in the study of a conjectured (so called AGT) duality between two quantum field theories defined 
on space of distinct dimensions: the four-dimensional Seiberg-Witten theory and the two-dimensional 
conformal field theory [33l|43]. They are also expected to play an important role in topological string 
theory and are used in the context of "quantum" Riemann surfaces [5j|8], which are parametrized by a 
formal Planck parameter h = (^]T/2 - y/2/]3)/N. 

If x = (xi, . . . , xjv) denotes the eigenvalues of a N x N random matrix X in the Jacobi /3-Ensemblc, 
we see that the integral (|1.1|) is proportional to the the expectation value of products of characteristic 
polynomials in X: 

K N (X 1 ,X 2 ,X;t 1 ,...,t n ) = S N (X 1 ,X 2 ,X) /f[dst(-tj+X)A (1.8) 

«' =1 ' XEJ/3E 

Similar expressions exist for the Hermite and Laguerre /3-Ensembles. This means that the expectation 
values of product of characteristic polynomials in /3-Ensembles of random matrices can be given in terms of 
hypergeometric functions involving Jack polynomials. Other probabilistic quantities such as the cumulation 
distribution of the largest or smallest eigenvalue, the marginal probability distribution of k eigenvalues 
have been related to the hypergeometric functions studied by Kaneko and Yan or the their associated 
classical symmetric polynomials [2|9l[TTl[T3l[T4j[26] . Despite the fact that the generalized hypergeometric 
functions are defined as sums over an infinite number of Jack polynomials, efficient algorithms have been 
found that make possible their numerical evaluation [26j . 

Although interesting in their own right, expectations of product of characteristic polynomial such as 
are not the most useful probabilistic object. Expectation of ratios of characteristic polynomials such 
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as 

9 (X A (X X X) / nLidet(l-t fc X)^ \ 

are often preferred since they are of particular interest in the study of the zeros of the Riemann £- 
function [23] and provide a method for computing the marginal densities of the eigenvalues also known 
as the n-point correlation functions (see for instance Proposition 2.16 in [7]). For the three classical cases 
(3 = 1,2,4, the exact calculation of these expectations have been completed by several authors (see [3J[7] 
and references therein). In the case where j3 is a general positive integer however, almost nothing is known. 



1.3. Goals and main results. Our aim is to exactly calculate the expectation ([1.8)1 of ratios of character- 
istic polynomials in the Jacobi-/3-Enscmble for /3 general, which is equivalent to the following generalization 
of the Selberg integral with A = /?/2: 



S N (\ 1 ,\2,\;t;s)= j D XM {x) ^- 1 Xjtkr d N x, (1.10) 

where it is assumed that sg e C\ [1, co) for all £. By looking at similar integrals but with -Dai.a 2 .a replaced 
by the densities in ([1.5)1 , we also want to find exact expressions for the expectation ratios of characteristic 
polynomials in the Circular-, Gaussian- and Chiral- /3-Enscmblcs. For reasons that will become clear in 
the next section, we will always suppose 

fj,i = 1 and fi2 = A. (1-11) 

One of the few results on expectations of ratios of characteristic polynomials in /3-Enscmbles was the 
observation in [SJ Section 6] that such expectation for the Gaussian /3-Ensemble is an eigenfunction of a 
second order differential operator which can be interpreted as the Hamiltonian in the so-called deformed 
Calogero model. The latter belongs to a more general family of quantum Calogero-Sutherland models 
involving two sets of variables (particles) that were extensively studied by Sergeev and Veselov [40")l4"l] 
and more recently in )12j . They were also shown to be related to super-matrices and symmetric super- 
spaces [T7] as well as to "multispecies Calogero models" exhibiting particlcs-antiparticles and strong-weak 
dualities [I]. 

In Section [2l we provide a stronger result than [9j Section 6] by showing that the function defined by 
integral ([1.10)1 satisfies a holonomic system of n + m nonsymmctric differential equations. 

Theorem 1.1. Let 

a = X, a = -N, 6 = -JV + l-(l/A)(l + Ai), c = -27V + 2 - (1/A)(2 + Ai + A 2 ). (1.12) 
Then Sjv(Ai, A 2 , A; t\, . . . ,t n ; s\, ... , s m ) satisfies 

ti(1 _ (0 ^ + ( c _ (o + t + 1)( A|_ o! , F + I ± ^((i- t ,)g 

1 fc=l,fc/i v 

- (i - **)w) -Et^((i- U)%- + i(i - s k )f) = 0, (1.13) 

ot k J ^ U - s k \ dt % a ds k J 

d 2 F ( , , . 1.x \ dF . . ^ s k 



1 3 F / 1 \ dF 

- Sj( l- Sj) + ( c -(a + b + l-(l + -)) Sj )--(-a)abF- £ 

(' - - a - + E fid - + a - 



/or aZH = 1, . . . , n and j = 1, . . . , m. Moreover, it also satisfies the cancellation condition 

'OF 1 OF N 



= 0, Vi,j. (1.15) 
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The sum of all the above differential equation gives a single differential equation of the form C n , m F = 0, 
where £ n ,m is a linear combination of operators that previously appeared in the article |12j . The latter 
was concerned with the solutions of some deformed Calogero-Suthcrland models [4TH - t4"2"] whose study 

requires the use of the super- Jack polynomials, here denoted by SP« (t; s). These objects were introduced 
by Kerov, Okounkov and Olshanski [24], and subsequently studied by Scrgeev and Veselov [41]. They 
generalize both the usual Jack polynomials P^it) (case with s = (0, ... ,0)) and supersymmetric Schur 
polynomials s K (t\s) (case a = 1). The latter were first defined as characters in the representation theory 
of the supcralgebra gl(n\m); see examples 23-24 in Section 1.3 in [29] and the article [34] . 

A solution for the equation for C n>m F = was given in j!2) as a super-hypergeometric function 

2<SF] {fli b] c;t\s), which can be written as an infinite sum of super- Jack polynomials. This provides 
us with one possible solution of the system (|1.13l) - (|1.14p . We will confirm this claim in Section^] Unfortu- 
nately, there is no uniqueness theorem for the equation C n-m F = in |12) that would allow us to conclude 
that the deformed Selberg integral in ([l.lOjl is indeed equal to 2SF[ a \a,b;c]t; s). As we will explain in 
Section [4] the method developed by Muirhcad and Kancko for proving the uniqueness for systems like 
(|1.4p does not extend directly for our case. 

We will tackle the delicate problem of uniqueness for the system (|1.13[) - (|1.14[) in several steps (Sections 
14.11 to |4~3]) . We will first prove that there is a unique symmetric formal power series that satisfies the 
infinite version of the system (|1.13[) (i.e., with n = oo). This will allow us to conclude that the solution of 
the system (|1.13j) - (|1.14j) . when both n and m are infinite. We will also show that the latter infinite system 
is compatible with the rectiction of the number of variable to the finite system. We will finally use this 
property and the fact that the super-hypergeometric function 2 SF^ a \a, b; c; t; s) satisfies some additional 
constraints, here called the Kaneko's criterion, to prove the following. 

Theorem 1.2. The function 2SF^ a '(a,b;c;t;s) is the unique solution of the system (|1.13[) - (|1.14[) subject 
to the following additional conditions: 



In Section [5] we will obtain more properties related to the super-hypergeometric functions p SF q . This 
will make possible in Section [5] to show that the expectations of ratios of characteristic polynomials in the 
Gaussian-, Laguerre-, and Circular- /3-Enscmbles are also super-hypergeometric functions. 

Finally, it is worth mentioning that the super- Jack polynomials have already appeared in the study of the 
Circular-/3-Ensemble. Indeed, the large N limit calculation of n-point correlation function for this ensemble 
was attempted in the preprint |37j and involved super series of hypergeometric type. More recently in |31j , 
some formal results on the ratios of characteristic polynomials in the Circular-/3-Ensemble were derived 
by using super- Jack polynomials and super-series. The latter preprint also contains interesting dualities 
relations between expectation values in the Circular j3 and 4//3 Ensembles. Such j3 -0- 4//3 dualities are 
common in Random Matrix Theory (see [9] and references therein). For more general ensembles, such as the 
Jacobi-/3-Enscmble, the duality relations for the expectation values of ratios of characteristic polynomials 
are much more complicated than their circular counterparts, and will be the subject of a forthcoming 
article. 



(1) F(t; s) is symmetric both in t\, . . . ,t n and in si, . . . , s m ; 

(2) F(t; s) is analytic at (0, . . . ; 0, . . .) and such that F(0, . . . ; 0, . . .) = 1; 

(3) F(t; s) satisfies the cancellation condition 




(1.16) 



Corollary 1.3. Let a, a, b, and c satisfy the conditions in (|1.12[) . Then, 



S N (\ l7 \ 2 ,\;t;s) = Sn(\i, A 2 ,A) 2 SF 1 t '(a, b; c; t; s). 



(i 
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2. Deformed holonomic systems 



We want to prove Theorem 11.11 In order to facilitate the presentation of the demonstration, we first 
introduce some more notations based on the theory of supermatrices [101117] . 

2.1. Z2-graded notation. Let w = Wi , . . . , w n +m be an ordered set of variables equipped with the 
following Z2-grading: Wi is even if i < n and odd otherwise. We set 



W; 



ti, i < n 



P(i) 



Let also p denote the Z2-gradcd function on {1, . . . , n + m} defined by 

1, i < n; 
— 1/a, i > n. 

We see that if a = A and equation (|1.11|) is satisfied, then 

N n+m yiN p-rri _ f 

nn<'-.v fa ° n"'n-' I 

j=l 4=1 Uj=l Ilfcll 1 - XjSi)^ 

The Z2-grading allows moreover to recast the two types of differential operators given in Theorem 1 1 . 1 1 into 
a single type of deformed differential operators: 

d 2 d d 

Sd = piWi(l - w l )—^ + c- (a + b + PijWi- — 

aw~ owi owi 

n+m 



1 v-^ 1 Wj ( , .8 , . d \ . 



a PiPj w i - w j \ dwi dwj 

More general deformed operators will be considered in Section |U 

2.2. System for Sjv(Ai, A2, A; t; s). We now proceed to show that for one specific choice of parameters 
a, b, c, and a, the function SW(Ai, A2, A; t; s) defined in (jl.lOj) is an eigenfunction of each operator SCi 
defined above. The method presented here is in essence equivalent to Kancko's, which basically uses Stokes' 
Theorem and clever choices of functions to be integrated and derived. However, proceeding exactly like 
Kaneko would rapidly lead us to long and almost uncheckable lines of equations. For this reason, in the 
proof, we prefer to interpret (|1.10[) not only as N dimensional integral, but also in terms of the scalar 
product over the Hilbert space of symmetric functions in L 2 (£l;p), where f2 = (0,1)^ and p is the 
absolutely continuous measure given by the Sclbcrg density (|1.2[) : 

dp{x) = f A A' Aa ' A(a l <fe, Ax,A 2 ,A>0. 

In what follows, the expectation value of any /i-intcgrablc function / will be denoted by brackets: 

(/)= [ f(x)dp(x). 
Jn 

Lemma 2.1. Let h : [0, 1] — > C be rational and regular. Define the following linear differential operator 
acting on smooth functions of J4?n : 

$> = J2( d ^+ V *( x )) h ( x *)> V <( x ) = d Xi (\nD XlMiX )(x), (2.2) 

i 

Then for any symmetric f : £1 — > C that is rational and regular, 

W)) = / 9(f)dft{x) = 0. (2.3) 
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Proof. Let us first recall that the weighted scalar product over Jftfo is given by 



(/Iff) = / J{x)g{x)d^{x), 
Jo. 

where the bar denotes the complex conjugation. This means that expectation value of / over O is equal 
to (/} = (/ 1 1). We now turn our attention to the domain A of the operator @. Given that 

Vi(x) = — 



Xi 1 Xi . . Xi Xj 

we choose A to be the set of all smooth and symmetric functions / in L 2 (il; /j,) such that f/xi G L 2 (fl; fi) 
and //(l — Xi) G L 2 (tt; fi). Note that S> preserves the symmetry of the functions /. so we indeed have 
A C J4?n ■ Since Ai,A2 > 0, the domain A includes all the functions we are concerned with, namely the 
rational symmetric functions that are continuous all over the closure fl of CI. Moreover, the condition 
Ai, A2 > implies that D\ lt \ 2i x(%) — for all x in dil = Q \ O. Hence, 



( y f(x)g{x)D Xu x^\{x)j dx = 0, Vi. 



This allows us to conclude that for all / and g in A , 

(?f\g) = (f\S>*g) with 9* = -Y,Wi)d Xi . 

i 

Consequently, for any / in A, we have (&(f)\l) = {f\S>*(l)) =0. □ 

Theorem 2.2. Let F(w) := Sn(Xi, X2, A; t; s) be the function defined in (jl.101) with Sj G C \ [l,oo) and 
Ai, A2, A > 0. Let also SCi be the deformed differential operator given in (|2.1| and assume 

a = X, a=-N, b= -N + 1 - (1/A)(1 + Ai), c = -2N + 2 - (1/A)(2 + Ai + A 2 ). (2.4) 

Then F(w) satisfies the following deformed holonomic system of partial differential equations, 

PiSCi F(w) — ab F(w), i = l,...,n + m, (2-5) 

and the cancellation property 

dF dF \ 

J / Wi—Wj 

whenever Wi and Wj do not share the same parity. 

Proof. Before showing anything, we note that F(w) is simply equal to the expectation (IT), where 



/Ui, i = l,...,n; 
-/i 2 , i = n + 1, . . . , n + Tn. 



3 i 

Also, assuming that \ii > and Wi G C\ [1, cxd) for all i > n, we remark that all the partial derivatives of 
IT with respect to the variables w belong to C°°(f2). This allows us to take the derivatives in w indifferently 
inside or outside the brackets (or equivalently, the integral signs). 

The first part of the proof consists in obtaining relations between expectation values involving IT. For 
this we successively apply Lemma [2.11 for different choices of operator SI . Let us set h(xi) = Xi in the 
definition (|2.2|) of the operator Qi and get the following operator: 

* ~ £ (9 Xi + KM) x t , Vi ( X ) = ±--^ + Y; 



According to Lemma [2~T1 we have («e/IT) = 0. Moreover 

9 Xi (n) = -n 



X% 1 Xi . . Xi 
37* 



<J,Wj 



L—' 1 — XsWi 
3 
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Simple manipulations then lead to 

».«-^(^|(^n).,. ,,7, 

where A 3 = N(l + X 1 + X 2 + X(N - 1)). Similarly, set h(xi) =1-Xi in (J272]) and define 

i 

From (gfll) = we get 

ajWj{l - Xi). 



We need to introduce yet another differential operator: 

= I] (3* + Vt(x)) (l - mm)- 1 , 

which corresponds to the operator 9) of (|2.2[) with /i(a;j) = 1/(1 — XiW k )- According to Lemma [2~T 
(^ fc n> = 0. This yields 

/ y- w k (l -a k ) \ _ / y> w^i n \ 

\^{l- Xi W k ) 2 J \,j^ k {l - X iWk ){\ - XiW/,) J 



E ' i_^ +E JL „L , (2 . 9) 

^ 1 - x l w k \ Xi 1- Xi Xi - xj I I 



After some simplifications, the substitution of (|2.7p and (|2.8I) gives 



1 / _a^£^ n \ _ lx^( 1 1 \ n \ 

1 - tUfe ^ \^ 1 - a: l W£ / JT^Wk - \ - XiWk 1 - a* to* / j 

-^(n) + (<x fe + A 1 + -^)/^-J— n\ = o. (2.io) 

1 — u>k V 1 — WkJ y-r'l-XiWk I 

For the second part of the proof, we will rewrite the last equation as a differential operator in the 
variables w acting on (II) . Little work first gives 

<n> = -a k (j2 r^^ n ) ' ( 2 - n ) 

dw k X^l-XiWk 



d 

w k - 



(IL) = a k N{IL)-a k (y2—^ H ) , (2.12) 



' dw k \ ^ 1 - Xjiu fe 

and 

a 2 



<9u> 2 (1 - XiW k ) 2 j (1 - XiW k )(l - XjW k ) 

Then, by making use of the identity 

XiXjwf. 1 11 



(1 - XiW k )(l — XjWk) (1 - XiW k )(l - XjW k ) 1 - X l W k 1 - XjWk 
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and of (|2.1ip - ()2.13|) . one can prove 

— (wkTTS + 2(1 - (TkN)wk^- - o k N(l - a k N) ) (n) 
<r k \ K dw% dw k 



a - *») (E 7i3^ n ) + °» ( E fl _ J a _ X < W A ^ 



i-^fe) 2 / (1 - XiW k )(l - XjWk) 

The last equation is proportional for every k to the two first terms of (|2.f Op if <Tfc(l — a k ) = — A(l — <7fc) 
As expected, the latter equation has two solutions <r k = 1 and a k = — A. We choose 



I 1, i = l,...,n; 

I— A, i = n + 1, . . . , n + m. 

This allows us to combine (12.101) and (|2.14[) and get 

— ( w l^~2 + 2(1 - <TkN)w k ^- - a k N(l - a k N)) (U) 
a k \ "dw 2 k dw k J 

,1 <Jk / Xiaiwe \ 1 ^ o k a f w k / ^ / 1 1 



(2.15) 



E/ Xj(Tew e \ _ 1_ (J k Qiw k / ST^ ( 1 



Al-wj; Y\ 1_ / ^TZi w k— We X*—' \1 - XiW k 1 - a^io* 

I \ i I £^k \ i x 



n 



-y^(n) + i(a fe + A 1 + -^)(E r ^n)=o. (2.16) 

A 1 - w k A \ 1 — ton,/ \ • £ r' 1 — x l w/ £ / 

Moreover, by substituting (|2.11[) and (|2.12p into the last equation, we actually find that (II) satisfies the 
desired equation as long as <Xj = 1/ pi, which imposes A = a, and the parameters a, 6, c are given by (|2.4p . 
Finally, one easily verifies that the cancellation property follows from (|2.1ip and (|2.15p . □ 

2.3. System for contour integrals. In the last theorem, the conditions on the parameters Ai, A2 and A 
were crucial for ensuring the convergence of the integral defined in (|1.10p and its derivatives with respect 
to the graded variables w k . It is possible however to interpret differently the integral without affecting 
the holonomic system it satisfies. For instance, in the case where Ai is a negative integer and A is a 
positive integer, then we can replace the volume integration over [0, 1]^ by repeated contour integrals such 
that each variables xj follows a closed loop in the complex plane starting at 1 and encircling in the 
counterclockwise direction. When A2 is not an integer, a branch cut is made from 1 to 00. This defines a 
new analytic function at w = (0, . . . , 0): 

r(0 + ) r(0+) N n + m 

T N (\ u \ 2 ,\;w) = / ... / ' Y[(l-x j w i ) 1 ^ i D Xl! x 2tX (x)dx 1 ---dx N , Ai e Z_, (2.17) 

*' 1 ^ l j=i 1=1 

where it is understood that KA2 > and A € N. In principle, the integral can be evaluated by using the 
residue theorem. 

What should be stressed here about the multiple integrals defining Tn is the following: (1) all the 
partial derivatives in w k commute with the integral signs; and (2) the integral of any "total derivative" is 
zero, i.e., 

/(0+) o N n+m 
d~\Ii II 0- - " "a-. x,.Ai.<-;) </■<-, = 0. 
Xl 3=1 »=i 

These two conditions are enough to guaranty that all the steps in the proof of the last theorem remain 
valid, so wc conclude that T/v also satisfies the same deformed holonomic system as Sjv(A, Ai, A2). 
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For similar reasons, the following contour integrals defines another function that is analytic at w = 
(0, . . . , 0) and satisfies the same deformed holonomic system: 

(1 + ) (i+) N n+m 

U N (X 1 ,X 2 ,X;w) = / •■• / l(l-x j w i ) 1 / pi D XuX2iX (x)dx 1 ---dx N , A 2 gZ_, (2.18) 

where it is also understood that 9?Ai > and A 6 N. If Ai is not an integer, we consider that each a^-plane 
is cut along (— oo,0]. 

2.4. System for Kn(Xi, A2, A; t; s). Theorem 12.21 does not apply directly to Kaneko-type integrals, such 
as (|l.lj) . There is however a very simple way to determine the deformed holonomic system related to such 
integrals. It relies on the following lemmas, which are easy to prove. 

Lemma 2.3. Suppose that G{w\, . . . , w m + n ) is one solution of the deformed system 

p,SC\ aAc) F{w) = abF(w). 

Then Yi^™ w i a ^ Pl G(l/wi, . . . , l/to ra+ „) is a solution of the following deformed system: 

Pi s4 a '> b '' c ' ) F(w) = a'b'F(w), 

where 

a = a, b' = a — c+ 1 + (l/a)(n — am — 1), c' = a — b + 1 + (l/a)(n — am — 1). 
Lemma 2.4. Suppose that G{w\, . . . , w rn + n ) satisfies the following cancellation property: 

dF _ dF \ 
% dwi ^dwj) 



Wi — Wi 



whenever Wi and Wj do not share the same parity. Then w i a ^ Pl G(l/u>i, . . . , l/w m + n ) satisfies the 

same cancellation property. 

Now, let us define an integral that generalizes both the fi = 1 and /i = —A Kancko integrals: 

r N FT- ix- -t ) 

K N (X 1 ,X 2 ,X;t;s) = ] tt™~) J -^D Xl . X2 ^(x)dx (2.19) 

where it is assumed that S" [0, 1]. If moreover tj 7^ 0, then we have 



K N (X X , A 2 , A; t; s) = (-l)C»- W f[ tf f[ s^ NX S N (x 1 ,X 2 , A; I . . . , i; . . . , — ) 

7 — 1 k— 1 x 7 



(2.20) 



In other words, K N (Xi, A 2 , A; w) = (-l)( n ~ Xm ) N Y[ i wl" ,,pi S N (Ai, A 2 , A; 1/w) with a = -N and a = A, so 
we can apply Lemmas I2.3H2.4I to Theorem [ 



Theorem 2.5. Let F(w) := K^(Xi, X 2 , X;t; s) be the function defined in (|2.19[) with Sf. G C\ [0, 1], tj ^ 
and Ai, A 2 , X > 0. Assume 

a = X, a = -N, b= (N - m- 1) + (l/A)(Ai + A 2 + n + 1), c = -m + (1/A)(n + Ai). (2.21) 

Then F(w) satisfies the following deformed holonomic system of partial differential equations, 

(p t S£i-ab)F(w) = 0, i = 1, . . . , n + m, (2.22) 

and the cancellation property 

dF dF \ 

L J / W-i—Wj 

where it is understood that Wi and Wj do not share the same parity. 
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In order to extend the range of possible values for the variables Sj, and allow for instance Sk G [0, 1), 
the contours of integration must be modified. Let ^ denote a closed path in the complex plane starting 
at 1, encircling the m variables Sj in the positive direction, and returning to 1. Suppose A2,A e N and 
Ai g' Z (for instance we can suppose that Ai has a non zero imaginary part). Then we define 

1 f f N IT- (x- —t-) 

K%(\ u A 2j A; t; s) = (e27ri(Ai _ Am) _ 1)N J^ -jJl U £^ - ^ ^.^(^1 ' " ' (2-24) 

The function iff (Ai, A 2 , A; *; 0, . . . , 0) is in fact an analytic continuation of the usual Kaneko integral 
-Kjv(Ai— Am, A 2 , A; t) with fx = l. Indeed, suppose as before A 2 , A G N, Ai Z, and additionally KAi— Am > 
0. Then, using the classical methods of complex analysis (such as those related to Hankel's contour for the 
Gamma function or Pochhammer's contour for the Beta function; see also [5J section 5]), we conclude that 
each f(x, t)dxi, where f(x, t) denotes the integrand of (|2.24j) , is equal to ( e 27 "( A i- Am ) - 1) J f( Xj t)d,Xi. 

Corollary 2.6. Let F(w) := K% (Ai, A 2 , A; t; s) be the function defined in (|2.24|) with Sk ^ 1, A 2 , A e N, 
Ai Z. Assume 

a = X, a = -N, b=(N-m-l) + (l/X)(\i + \ 2 + n + l), c = -m + (1/A)(n + Ai). (2.25) 
Then F(w) satisfies the deformed holonomic system (|2.22j) and the cancellation property (|2.23|) . 

3. Supers ymmetric functions and associated hypergeometric series 

This section first provides a brief review of some aspects of symmetric polynomials and especially Jack 
polynomials. The classical references on the subject arc Macdonald's book [35] and Stanley's article |44) . 
This will allow us to introduce the supcrsymmetric Jack polynomials [41) and especially their associated 
hypergeometric functions [12]. A few results proved here will be used later in the article. 

We stress that in the following pages, we always assume F = Q(a), which means that F is equal to the 
field of rational function in the formal parameter a. 

3.1. Partitions. A partition k = (kx, k 2 , . . . , k,, . . .) is a sequence of non-negative integers Ki such that 

K-l > K2 > ■ ■ ■ > Hi > ■ ■ ■ 

and only a finite number of the terms Ki are non-zero. The number of non-zero terms is referred to as 
the length of k, and is denoted £(k). We shall not distinguish between two partitions that differ only by a 
string of zeros. The weight of a partition n is the sum 

\k\ := ki + k 2 + • • • 

of its parts, and its diagram is the set of points (i, j) <E N 2 such that 1 < j < Ki- Reflection in the diagonal 
produces the conjugate partition k! = (k[, k' 2 , . . .). 

Let k, a be partitions. We define k U a to be the partition whose parts are those of k and cr, arranged 
in descending order. For example, if n — (321) and a = (21), then k U a = (32211). 

The set of all partitions of a given weight are partially ordered by the dominance order: k < cr if and 
only if J2i=i K i ^ Ei=i °i f° r a H One easily verifies that k < cr if and only if a' < k'. We shall also 
require the inclusion order on the set of all partitions, defined by a C k if and only if cr, < Ki for all i, or 
equivalently, if and only if the diagram of a is contained in that of k. 

3.2. Jack symmetric functions. Let A^(x) denote the algebra of symmetric polynomials in N indeter- 
minates x\, . . . , xn. As a ring, it is generated by the power sums: 

p k (x) := x\ + ... + x h N . 

The ring of symmetric polynomials is naturally graded: An(x) = (Sk>o-A l ^(x), where A^(x) denotes the 
sets of homogeneous polynomials of degree k. As a vector space, A^(x) is equal to the span over F of all 
symmetric monomials m K (x), where k is a partition of weight k and 

m K (x) := x^ 1 ■ ■ ■ x^ N + distinct permutations. 
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Note that if the length of the partition k is larger than TV, we set m K (x) = 0. 

The whole ring A jy (x) is invariant under the action of homogeneous differential operators related to the 
Calogero-Suthcrland models [5] : 

N r) N f) 2 9 T k r) 

f-f ox,; f-f dxf a ' Xj - Xj dxi 

The operators i? 1 and TO 2 are special since they also preserve each A k N (x). They can be used to define the 
Jack polynomials. Indeed, for each partition k, there exists a unique symmetric polynomial (x) that 
satisfies the the two following conditions |44j : 

(1) P^ a) {x) = m«(x) + c K „m,,(x) (triangularity) (3.2) 

(2) (d 2 - i(JV - l^J Pi a) (x) = e K p( a) (x) (eigenfunction) (3.3) 

where the coefficients e K and c KM belong to F. As a condition of the tiangularity condition, Ajv(x) is equal 
to the span over F of all Jack polynomials Pk°\x), with k a partition of length less or equal to TV. 

Following the standard nomenclature of algebraic combinatorics [55] , when the number TV of indctermi- 
nates is infinite, we say "symmetric functions" instead of "symmetric polynomials" (although in general 
the "functions" cannot be considered as mapping from a set to another) . The ring of symmetric functions 
is given by A = ©fc>oA fc , where A fc is equal to the inverse (or projective) limit of A^. In other words, if 
z = (zi, z 2l . . .) stands for an infinite set of intcderminates, each element of A(z) is equal to a finite linear 
combination of monomial symmetric functions m K (z) with coefficients in F, where 

m K (z) = (m K (zi),m K (zi, z 2 ),m K (zi, z 2 , z 3 ), . . .). 

The addition and the multiplication in A (z) acts component by component. Similarly, we can define the 
Jack symmetric function P^ a \z) as an inverse limit of polynomials: 

P^(z) = (P^(z 1 ),Pi a \z 1 ,z 2 ),P^(z u z 2 ,z 3 ),...). (3.4) 

This object is well defined since the coefficients of the Jack polynomials are stable with respect to the 
number N of variables: the coefficients c KM in (|3.2|) do not depend upon TV. The power sum polynomials 
also share this stability, so we can form the fc-th power sum function pk(z) similarly. Thus, the algebra 
A(z) of symmetric functions can be seen as the F-span of either {m K (z)} K , {Pk (z)} K , or {p K (z)} K , where 
k runs through the set of all partitions and 

p K (x) :=p Kl (z) ■ ■ -p Kt (z). 

3.3. Super-symmetric functions. Let z = (z\, z 2 , ■ ■ .), x = (x\, x 2 , . . .) and y = (j/i, y 2 , . . .) be three 
sequences of independent indeterminates. As explained above, A(z) is the graded algebra consisting of 
symmetric functions over F. The algebra of bi-symmetric functions is given by A(x) <8> A(y), that is, each 
element f(x,y) of this algebra is symmetric in x and y separately. 

Now let 5A(x, y) be the subalgebra of A(x) ® A(y) formed by the bi-symmetric functions /(x, y) that 
satisfy the following cancellation conditions: 



+ 

dxi a dyj 



0,Vi,j GN. (3.5) 



Each element of SA(x, y) is called a supersymmetric function. It is very easy to check that the homogeneous 
elements of 5A(x, y) also satisfy p.5[) , so we have the grading SA = (Bk>o SA k . Moreover, there is a natural 
ring homomorphism tp : A(z) — > SA(x,y) such that 

V(pk(z)) =Pk,a(x,y) :=Pk(x) -OLPk(y). (3.6) 
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Such a homomorphism was first used by Kerov et al. in [24]. We call pk, a {x,y) the fc-th deformed power 
sum (in infinitely many variables). Let Af a (x,y) denote the algebra generated by these power sums. In 
other words, 

M a {x 7 y) = Span F {p KiCt (x,2/)} K 

where 

p K ,a{x,y) = Y[ Vm,ot{x,y)- 

Ki>0 

Obviously, we have M a (x,y) C SA(x,y). As proved below, the algebra of supersymmetric functions 
coincide with the algebra Af a (x,y) for specific choices of the parameter a and in particular, for all real 
a > 0, which is the relevant case for the study of the deformed Sclbcrg integral . 

Lemma 3.1. If a is neither a negative rational number nor zero, then SA(x, y) is generated by the deformed 
power sums p r , a (x, y), r G N. 

Proof. Since the p r {x), with r > 1, are algebraically independent over F, so are the p r , a {x, y) of Af a (x, y). 
The mapping if defined in (|3.6p is then an isomorphism between A(z) and Af a {x, y). As we have already 
pointed out Af a (x,y) C SA(x,y). Thus, to show they are actually the same it is sufficient to prove that 
the dimension of the homogeneous component of degree k of Af a (x,y) is not greater than the number of 
partitions K of k. 

For n,m G N = {0,1,2,...}, set x = (xi,...,x n ) and y = (yi,...,y m ). Let us denote by SA n>m (x, y) 
the subalgebra of A n (x) ® A m (y) consisting of bi-symmctric polynomials f n ,m(x,y) which satisfy the 
cancellation conditions. For given k, taking n, m > k, we define a homomorphism (f> nm from SA(x, y) to 
SA n . m (x, y) by setting x n+1 = ■ ■ ■ = 0, y m+1 = ■ ■ ■ = 0, that is, 

<t>n, m {f) = f(xi,...,x n ,yi,...,y n ), VfeSA(x,y). (3.7) 

Since the homogeneous components of degree k of the rings A(a;) (8) A(y) and A n (x) (£> A m (y) are isomor- 
phic, the homomorphism </>„ im , once restricted to the homogeneous components of degree k, is injective 
whenever k < min(n, m). When a is neither a negative rational number nor zero, by Proposition 2 of |40j . 
the dimension of the homogeneous component of degree k of SA n ^ n (x,y) is not greater than the number 
of partitions k of k, which implies that it is also true for SA(x, y). □ 

Let us go back to the homomorphism (|3.7p . which restricts the number of indeterminates. If we combine 
with the map tp of (|3.6[) . we obtain a homomorphism between the symmetric functions and the super- 
symmetric polynomials: 

o ip : A(z) — > SA n . m (x, y). (3.8) 

The image of a Jack symmetric function, which is defined by (|3.2p - (|3.3[) and (|3.4p . under the action of 
(p n ,m is called a super Jack polynomial [3"Tll41j . We write 

SP^(x u ...,x n ;y u ..., y m ) = ip n , m (P^(z)). (3.9) 

Likewise, we call the image of a Jack symmetric function under ip a super Jack symmetric function. 

Let H n ,m be the set of partitions which consists of the partitions k such that K n +i < m. In other words, 
k S H n ^ m if its diagram does not contain the square (n + 1, m + 1). Recently, Sergeev and Veselov have 
proved the following very important properties of super Jack polynomials (Theorem 2, Til): If a is neither 
a negative rational number nor zero, then the kernel of ip n ,m is spanned by the Jack symmetric functions 
Pk"\z) with (n + l,m + 1) G k. Moreover, the polynomials 

SP( a >(xi,...,x n ;y 1 ,...,y n ), k g H n , m , 

form a linear basis of 5A„ )m . 

In the present article we will treat our objects in the super-symmetric space 5A(or SA„ !m ) or as the 
associated super series given below, in particular when a > 0. 
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3.4. Super hypergeometric series. Recall that the arm-lengths and leg-lengths of the box in the 
partition k are respectively given by 

a K (i,j) = Ki- j and = Kj - i- (3.10) 

We define the hook- length of a partition k as the following product: 

J#° = II {} + a K (i,j) + U K (i,3))- (3.H) 
Closely related is the following a-deformation of the Pochhammer symbol: 



mL 0) = 



II { x - l -^) K . = II (x + a' K {i,j)-U' K {i,3)) (3-12) 

l<i<f(n) Kz {i,j)EK 



In the middle of the last equation, (x) n = x(x + 1) • ■ • (x + n — 1) stands for the ordinary Pochhammer 
symbol, to which [x]k clearly reduces for £(n) = 1. The left-hand side of (|3.12[) involves the co-arm-lengths 
and co-lcg-lengths box (i,j) in the partition k, which are respectively defined as 

a' K (i,j) = j -1, and l' K (i,j)=i-l. (3.13) 

By looking at the formulas above, one readily proves the following. 

Lemma 3.2. Let M be a positive integer. Let k be a partition of length £ and first part equal to k±. 
Then, for all a > 0, [-M}[ a) ^ if Ki < M, while [~M}{ a) = if Ki > M. Moreover, for all a > 0, 
[M/a]l a) ^0if£< M, while [M/a]^ = if £ > M 

We are now ready to give the precise definition of the super hypergeometric series in question. 

Definition 3.3. Fixp, q € No and let a, . . . , a p , b\, . . . , b q be complex numbers such that (i— l)/a— bj ^ No 
for all i G No- We then define the (p, q)-type super hypergeometric series by the following formal power 
series: 



p flFW (a lf • • • , a p ; b u . . . , b q ; *, „) = ^ ^ " a ^ ' " ^ SP^(x; y). (3.14) 



When we are given finitely many variables, x = {x\, . . . , x n ) and y = (yi, . . . , y m ), the above series 
coincide with the super hypergeometric functions introduced in [12] . The latter were given as the image of 
formal power series in Jack symmetric functions under the action of the restriction homomorphism ip n , m - 
If we suppose moreover that y = (0, ...,0), then we recover the (generalised) hypergeometric functions 
studied, in particular, by Koranyi [37], Yan |3B], and Kaneko pSFq will be denoted by p Fq a ^ 

whenever y = (0, 0, . . .). 

Some peculiarities of the series p SF q with n+m variables deserve to be mentioned. First, there is always 
an infinite number of terms in the series (|3.14l) that do not contribute to p SF q since SP K (x,y) = unless 
k G H n ,ra- Second, the super series p SF q does not terminate as easily as the usual p F q . To understand this 
point suppose that in (|3. 14[) . we have y = (0, . . . , 0), q > 1, and a\ = — M for some integer M. According 
to Lemma 13.21 no partition with k\ > M will contribute to the series. Moreover, the Jack polynomials 
P K {x\, . . . ,x n ) = for all partitions k with length £ > n. This means that for the case with some ai equal 
to a negative integer, the usual hypergeometric series p F q is actually a polynomial. However, a super Jack 
polynomial SP K {x\ , . . . , x n , y\, . . . , y m ) ^ even if the £(k) > n + m, so the series p SF q with some ai equal 
to a negative integer does not terminate in general. 

The following proposition shows that the integral formulas relating p SF q and p +\SF q +i series also have 
to be treated slightly differently than for the usual hypergeometric series in many variables, which are due 
to Yan for general a > |46j . These new formulas are expressed in terms of series involving both Jack 



SELBERG INTEGRALS AND SUPER HYPERGEOMETRIC FUNCTIONS 



15 



and super Jack polynomials: 

pSF^ (a 1 ,...,a p ;b 1 ,...,b q ;z 1 ,..., z e ; x, y) 

Note that the above sums makes sense only if it extends over all partitions length less of equal to 



(3.15) 



Proposition 3.4. Let a\, . . . , a p +i, bi, . . . , b q +\ be complex numbers such that (i — l)/a — bj £ No for all 

i 6 No. Suppose moreover that for some positive integer I and some 1 < i < p + 1, we have 

£ - 1 £ - 1 I 

a v+i = Ai + 1 H , b„ + i = Ai + A2 + 2 + 2 and <Xi = — . 

a a a 

Then, formally 

p+1 SF^\(a 1 ,. ..,a p ;bi,...,bg',x, y) 

p ST q a) (ai, . . . , a p ; b x , . . . , b q ; z x , . . . , z e ; x, y)D Xu x 2A/a (z 1 , z e )d e x. (3.16) 



c / q 

*Ai,A2,l/a J[0S] e 

Proof. Here we treat p SF q as a formal power series, so we suppose that it can be integrated term by term. 
In fact, the proposition is almost a direct consequence of Kadell's integral formula [20j|21] (see (12.143) 
in|H|): 



Pj: a) (xi, . ..,x t ) D x X 2,l/a(xi, ■ ■ .,x t )dx 

bi{\\, A 2 , l/a) J[ 0tl y 

= pW(iQ [Ai + i + e-D/q]^ (3 . 17) 

[A 1 +A 2 + 2 + 2(f-l)/a]L Q) 

We see that the integration of P^ a \x) / P!f"\l e ) precisely gives the factors [a p+ i]^ and [b q+ i]^ that 
should appear in p+ iSF q +\. Given that the series p «SFg does not contain terms associated to partitions 



of length greater than £, the integrated series can not contain such terms either. Recalling Lemma 13.21 
we conclude that the integrated series is indeed equal to p +x<SFi°i if at least one the parameters a, of the 
latter series is equal to £/a. □ 

In the subsequent sections, we will pay special attention to the hypergeometric series qSFq, iSFq, 0SF1, i<SFi 
and 2SF1. As for the classical Gaussian hypergeometric function, these series satisfy simple systems of 
differential equations of second order. 

4. Existence and uniqueness theorems for holonomic systems 

The aim of this section is to establish existence and uniqueness for solution of the deformed holonomic 
system with finitely or infinitely many variables. We will prove that the super hypergeometric series 
2SFi a \a, b; c; t, s) is the desired solution. 

4.1. Holonomic system with infinitely many variables. In this subsection we study the special case 
of the system (|1.13[) and (|1.14[) with n = 00 and m = 0, which corresponds to a non-deformed system but 
with infinitely many variables: 



d 2 F / \dF 

W(! - Vi)gfi + [ C - ( a + b + l )Vi) dy~ - abF 



1 Vk (, ,dF , 8F\ , N 

- E — =0, i = l,2,.... 4.1 
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The finite case n < oo is already well understood. Indeed, by generalizing Muirhead's result |36j for 
the case a = 2, Kaneko [3T] and Yan [JB] independently showed that 2-^1 ( a , b; c; j/i, . . . , y n ) is the unique 
solution F(yi, . . . , y n ) to the following system that is symmetric in yi, ■ ■ ■ ,y n and analytic at (0, . . . , 0) 
with value F(0, ... ,0) = 1: 

Q2 j? / 1 1 \ QF 

Vi{l - Vi)^r + [c- -(n - 1) - (a + b+ 1 - -(n - l))w)^ a&F 

dyf \ a a ' J dyi 

1 " 1 / dF OF \ 

+ - E yKi-yOT^-y fe (i-2/fc)^H < i = i,...,n. (4.2) 



The systems (|4.1|) and (J4T2J) are easily shown to be equivalent when the number of variables is finite. 
However, in the infinite case, only the former makes sense. 

We first demonstrate the following uniqueness property. Its proof can be given in a similar way to that 
of the finite case by Muirhead [35] and Kaneko [21] . 

Theorem 4.1. Assume that c — (l/a){i — 1) is neither negative integers nor zero for any i £ N. Then 
each of the infinitely many differential equations in the system J^.i| ) has the same unique formal power 
series solution F(y) subject to the conditions 

(a) F is a symmetric function of {yi}°Zi> an d 

(b) F has a formal power series expansion at y = (0, 0, . . .) with F(0, . . .) = 1. 

Proof. We first transform (|4.1|) to partial differential equations in terms of elementary symmetric functions 
{rj} C jZ 1 , which are defined as 

r 3 = E " ■ yi i ■ 

ii<...<ij 

Let Tj denote the jth elementary symmetric function formed from the variables {yj}j^i\{yi}- Then we 
clearly have 

r 3 = wf^ + rf , j = 1, 2, ... , with r = r$ = 1. (4.3) 

Given that 

doc a o 00 o2 

_-Vr W — and — - V r W r W — 

% " ^ V ~ X dr v and dy\ ~ ^ Vi r -i dr drv ' 

we easily see that the differential equation in f|4.ip is equal to 



dr^dr u 



E y^ 1 ~ y*) r ll-i r v-i 



OC 1 OC / v 

E{( c - ( a + b + + 7: E r^r ( x - y*>ti - a - w)r£i } 



fe=l J=l 



dF 

dr k 



abF = 0. (4.4) 



Now, for i =/= j, let us denote by the fcth elementary symmetric function in the variables {yk}kLi\{yi, Vj}- 
We set = if k < 0. By successively making use of the relation (|4.3[) . we get 

E2/i /Oi \ (<) H \ (j) \ Vi ( (i) (j) , (t) 

— ((l-w)rU l -(l-wMii)= L — K-i-^-i+^ '-riM 

OO / \ OC 

E w(r£2+r£S)=- E 

=l,.7^t V ' .7 = 1, .7= 



'fc-1 'fe-1 "r ' fc 'fc 
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Writing yj as yj — yi + yi, we then find 

t .^(c-^-c-^) 



E r«-r£5 + r»-r{W - £ 1^-^1+4' 
i=ij'#j j'=i,j'#« 



E - r£5 + r« - rf ^) = -(* - l)r« x - fcr«, 



where we have used 



E (r?-rF ) )=kr?. (4-5) 
i=i V J 

In fact, given a sequence i\ < i<z < • ■ ■ < ik such that i; 7^ i, I = 1,2, ... ,k, the factor y^y^ •••y% h 
occurs exactly k times in the summation of the left-hand side of (|4.5[) . This can be made more precise: 
ViiViz ■ ■ ■ Vi k docs not disappear in the term (r$ — r^' ) if and only if j = i t for some t G {1,2,..., k}. 
Hence, 



-(a + &+l)tt)r« 1 + i E -7^-((l-^i-(l-^i) 

(c - -(k - 1))^! + (a + 6 + 1 - -fc)r^ l) - (a + 6 + l)r*. (4.6) 



Further identities are needed for simplifying the differential equation (|4.4|) . We note that 

(i) W (») U) W 

(0 fOc (0 \ 

— r — r^r i ■+- i/r^M^ 

By iterating this relation, we get 

(0 (») (») (i) , (») CO 



00 (*) , («) (») 



and = 0. Furthermore, 

t-\ \ (0 (0 
2/i(l-2/iK-i^li = 



-2(^4.! - r v+2 + ry{ 2 ) + • • • + r^-i^ - n + r^) - ro^j^ - r„ +fi + r^) 

OO 

= E< ) (ril 1 -r fc +r«) J (4.7) 



fc=i 



where (see James [19], p. 372-373 for more details) ajS = and for /i < z/, 

r^+u-k for I < k < (i 



-r^+^-fe for 1/ < fc < /it + z/ 
for /i + 1/ < fc fc = 1, 2, 
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Therefore, according to (|4.6j) and ()4.7j) . the differential equation <|4.4|) can be rewritten as 

d 2 F 



fiM—l k—1 ^ 



]T{ (c - -(k - l))r^ j + (a + b + 1 - -k)rf - (a + b + - abF = 0. (4.9) 



fe=i 



(i) 

In the latter equation, we can equate coefficients of ri_i to zero for k = 1, 2, . . ., according to the following, 
which will be showed in Appendix [A] 

Lemma 4.2. Assume that {yj}JL 1 are infinitely many indeterminates . Let {rj} c *L 1 and {r^}JL 1 be the 
elementary symmetric functions of {yj}°° =1 and {yj}j°=i\{yi} , respectively. If Xo(r), Ai (r), . . . , are formal 
power series of {rj}°°^ 1 such that 



^X k (r)r^=0, (4.10) 



.(*) 

fc=0 

£/ien Afc(r) = /or fc = 0, 1, . . .. 

Equating the coefficients of r k _ 1 in (|4.9[) to zero, we obtain the system of partial differential equations 

£ + ^/Ir + (c - i(fc - + (a + 6 + 1 - i(* - i))^- 

flM—l h 



oo oo 



-**{E (E^)^r + ( a + b + 1 )E^'^: + a&F } = ' fc = 1 ' 2 '---- ( 4 - n ) 

^,i/=i j=i ,y j=i 3 

Note that by convention, a ®] = and j^f- = 0. 

Let £„ = {(ii, . . . , i n )|ii, . . . ,i„ £ No} and £ = U^Li £«• We introduce the lexicographic ordering on 
£, that is, for / = (i\, i 2 , •■•,), J = (ji, J2, ■ • • , ) 6 £, saying I < J if either I = J or there exists n £ N 
such that ifc — = for k > n but i„ — j n < 0. Now we write 

F(r) = J2^V (4-12) 

lea 

with 7(0, . . . , 0, . . .) = 1. Here r 1 = r^r 1 ^ ... is only a finite product since the components of the index / 
are zero except for the first finite. Order coefficients according to the lexicographic ordering of the 
index /. To determine the coefficient 7(^1, . . . , i n , 0, . . .) where i n > 0, substituting (|4.I2|) in (|4.Iip with 
k = n and putting rj = for j > n, we obtain a recurrence relation expresses i n (hi — 1 + c — (l/a)(n — 
l)j7(ii, . . . , i ni 0, . . .) in terms of coefficients of lower order. Since i n — 1 + c— (l/a)(n — I) is not zero by 
assumption, one can iterate this reduction until one reaches 7(0, . . . , 0, . . .). Hence all coefficients 7(1) of 
(|4.I2[) are uniquely determined by the recurrence relations. On the other hand, each of the infinitely many 
differential equations in (|4.9[) gives rise to the same system (|4.11[) . so it follows that each equation in the 
system (|4.ip has the same unique solution F subject to conditions (a) and (b). This completes the proof 
of Theorem O □ 

It remains to show that the hypergeometric series 2^"' (a, 6; c; y) actually satisfies the infinite holonomic 
system (|4.f p . 

Theorem 4.3. 2-^1 (o, b; c; y) is the unique solution of each of the infinitely many differential equations 
in the system J^._?[ ) subject to the conditions 

(a) F is a symmetric function of {yi}fZi> an d 

(b) F has a formal power series expansion at y = (0, 0, . . .) with F(0, . . .) = I.. 
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Proof. The series (|4.12j) can be rearranged as a series of Jack symmetric functions 



%) = EE i^ a \y). (4.13) 

k=0 \n\=k 



It suffices to show that 

_ [ai a) [bj a) 

For any partition a and given n > max{l, £(<r)}, putting yj = for j > n, F(yi, . . . , y n , 0, . . .) = 
J2kLo J2\ K \=k IkPk (yi, ■ ■ ■ 7 Vni 0, . . .) clearly satisfies the finite system (|4.2|) . The theorem finally fol- 
lows from a result due to Yan [461 Theorem 2.13]: There exists a unique sequence {A K } with = 1 
such that F n (yi, . . . ,y n ) = J2 K A K P K { a ) (yi, . . . , y n )/hn satisfies (|4.2|l for n = 2,3,.... Moreover, A K = 
[at\bt ] /[ct ] . □ 

By proceeding as in the proof of Theorem 14. 1[ one can get the following useful generalization. 

Theorem 4.4. Fix po E N U {oo} and let a\, a 2 , flo, f3i, 70 be complex numbers such that /3o — — j) a i 
are never zero for any 1 < i < pq, j G No- Then each of the countably many differential equations in the 
system 



d 2 F dF 

+ ctiyi)—^- + (/3 + PiVi)-^ V 7oF 

dyf dyi 



1 v 2 ^ Vk (, ,dF , ,dF\ , 

~ zl - J ^((ai+a 2 y i )—-(a 1 +a 2 y k ) 1 r-)=0 4.14 



k=\.k=£i 

has the same unique formal power series solution F(y) subject to the conditions 

(a) F is a symmetric function of {yi}^i, and 

(b) F has a formal power series expansion at y = (0, 0, . . .) such that F(0, 0, . . .) = 1. 
Corollary 4.5. Consider the system (|4.14[) when the parameters (ai, a 2 , /?0i , 7o) are evaluated at 

(l,0,c,-l,-a), (0,-l,l,-o-6-l,-o6), (l,0,c,0,-l), (0,0, 1,-1, -a), and (0, 0, 1, 0, -1). 
Then, the unique solutions of the system satisfying conditions (a)-(b) are respectively given by 
1 Fi a \a;qy), 2 F (a) (o,6;y), F^(c;y), ,F^\a;y), and F (a) (y). 

In particular, 

Po Po 
iF^\a;y) = Y[(l-y. l )- a and F^ (y) = J] ^- 

i=l i=l 

Proof. One simply specializes the values of ai, a 2 , /?o ; Pi, 7o and uses the confluence relations, such as 

lim 2 F[ a \a,b;c; -y) = x F[ a) (a;c;y) 

b — yoo 



and 



lim 1 F^ ) (a;c;-y)= Ft\c;y). 

a— >oo a 



The systems for iF^ and i-Fq Q ' are trivial and can be explicitly solved. □ 
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4.2. Deformed holonomic system with infinitely many variables. Here we want to show that 
the super hypergeomctric series 2 SF^ a \a,b; c;t,s) is one solution of the infinite deformed system, which 
corresponds to (| 1 . 13|) and (jf .f 4[) with n = oo and m = oo. 

We first introduce non-symmetric versions of the differential operators E l and D k , which naturally 
appear in the study of Calogcro-Sutherland 



E\ = z\ — , ieNo (4.15) 

OZi 

and 

Obviously, these operators do not preserve ring A(z) of symmetric function in the indctcrminates z = 
zi, 22) ■ • •, but as we will see below, their action is sufficiently "nice" to be exploited. Note that this new 
notation, the non deformed infinite system of Theorem 14.41 can be written as: 

&F = >y F, C i = a 1 Dl + a 2 D^+p Ef+PiEl i = 1, 2 (4.17) 

We then generalize the Z2-grading introduced in Section [5] by collecting the two sequences of indeter- 
minatcs x = (x\, x 2 , ■ ■ ■) and y — (yi, j/2, ■ • ■) into a single sequence w = (wi, w 2 , . . .), where 

X{i+i)/2, odd i 
Vi/2: even i 

Also, we introduce a map p : {1, 2, ...,}-> F by specifying the value of p(i) according to 

1, odd i 
-1/ct, even i 

As explained in Section [3l the algebra SA(x, y) of supersymmetric functions is generated by the deformed 
power sum (|3.6j) . which can now be written as 



Pi = P(i) = 



' 1 

PrA x 'V) = 52 — w i 
^ Pi 



Furthermore, we define the following deformations of the operators E\ and D 



i=l 



SE\=w\^-, (GNo (4.18) 
dwi 



and 



r,r^k k d 2 1 1 Wj ( fc-i d fc _i d 

SD " = P^7^ + 7 E 777777 77 ['''" • 177 Pi«? 



,k-i d 



(k-^il-p^wf- 1 — , fcGN. (4.19) 

OWi 



The infinite dimensional equivalent of the operator SCi defined in (|2.1[) , which reads 

d 2 d d 
SCi = PiWi(l - Wi)—^ + c- (a + b + pijWi- — 

OWJ OWi OWi 



+ -V ^—( P i(l - - Pi(l - ) , (4.20) 

can be simplified as 

Sd = SD\ - SD 2 + cSE? - (a + & + lJSE? . (4.21) 
By summing up all the operators SCi one obtains a deformed operator of Calogcro-Sutherland type that 
preserves the algebra SA — SA(x, y). However, the image of a super symmetric function under the action 
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of SCi is not super symmetric, so we need a bigger ring than SA. We first define A[zi, . . . , z n ] as the ring 
of polynomials in indctcrminates z\, . . . , z n with coefficients belonging to A = A(z), where n G N. Second, 
we form the direct limit 

A= |J A[zi,...,z„]. 

)i>i 

It is clear that E\ and Df map A into A. Likewise, SA is the direct limit of the rings SA[w\, . . . ,w n ] of 
polynomials in w\, . . . , w„ with coefheients in SA(w). The deformed operators SE\, SDf map SA into SA. 

Now we interpret both A and SA as vector spaces. This means that the homomorphisms of algebras 
tp : A — > SA, defined by tp{p r ) = Pr.a, is now seen as a linear map. We finally extend ip to the linear map 
tp : A — > SA as follows: 

for all "coefficients" / in A, i\ < ■ ■ ■ < i q , and ji, . . . , j q > 0. We stress that tp is not a ring homomorphism. 
Proposition 4.6. For all I G No, k,i G N, the following diagrams are commutative: 



A 




-+ A 




A 


D k 
— > 


A 


i 






and 


4 








SA 


se\ 


-+ SA 




SA 


SDi 
> 


SA 



(4.22) 



Proof. Note that E\ are first order differential operators and that A is generated by the power sums p r {z) 
with r G N, it is sufficient to compute their action on the power sums. Observe that 

tp(E l lPr ) = -w^- 1 = SE\p{p r ), 
Pi 

This yields the left-hand side of fljjgg) . 

We now focus on the differential operators D\ and SD*. It is sufficient to consider their action on the 
product Pr(z)p s (z) and p r ,a{w)Ps,a{w), respectively. Note that 

ffif(Pr,aMp., W) = 2rS-W* +r+S - 2 +p r<a {w)(SD k i PsAw))+PsAw){SD k ; PrAw)). 

Pi 

Moreover, direct calculations lead to 



fc+r-3 

r—2—t 



= r V \—Ph+r-i-t,a - — +r(fc + r-2)(l 

V ft ft / Pi p 



and 



Finally, the last few equations and the use of 



pivtisPrAv) = r ( r - l)^' +r " 2 = r(r - lp + r(r - 1)(1 - -)^ +r ~ 2 . 

0< p l Pi 



i-i) fi+J-i 

ft/ V "ft, 
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allow us to write 



1 7/j fc+r - 2 



SD k ( Pr , a (w)p s , a (w)) = 2rs-w k + r+s - 2 + r(r - lp Pa , a (w) 

pi Pi 

+ s(s— 1) — Pr,a(w) H <^r \^ ( — Pk+r-2-t,a 5 )Ps,a(w) 

Pi al f^o Pl Pi 

+ 5 — 3 + fc-l-e — 2 

+ S ( p fc+s _ 2 -t,t« \p r , a (w)>. 

t=0 Pl Pl 

When all pi are equal to 1, we see that the action of D\ on p r (z)p s (z) has the same form as above. 
Consequently, the commutativity of the diagram on the right-hand side of (|4.22|) holds true. □ 

As long as we are dealing with countably infinite sets of indeterminates, both linear maps <p and dp are 
injective and surjective. The existence and uniqueness of the solution for the infinite deformed system 
then follows from Theorem 14.31 and Proposition 14.61 

Theorem 4.7. 2SF1 (a, b; c; t, s) is the unique solution of the infinite deformed system defined by (| 1 . 1 3[) - 
(I1.14[) with n, m = 00, subject to the following conditions: 

(a) F is symmetric in t = (t\,t2, • • •) and s = (sx,S2, ...) separately and satisfies the cancellation 
condition (|1.16|) . 

(b) F has a formal power series expansion at (t, s) = (0, 0) with F(0, 0) = 1. 

Remark 4.8. If we let x — (xi, . . . , x n , 0, . . .) or y = (y\, . . . , y m , 0, . . .), the commutative diagrams as above 
still hold true. This immediately implies that the corresponding scries 2SF1 (ct, b; c; t\, . . . , t n , si, . . . , s m ) 
is one solution of the finite deformed system (|1.13|) - (|1.14|) . The uniqueness is not guarantied however. 



The combination of Theorem 11.11 with n, m = 00 and Theorem 14.71 gives us one of main results of the 
article. 



— r TT / I t-too n ^D Xu \ 2 ,x{xi,...,x N )d X 

1, A 2 ,A) J, ii« ~~ 1 lfe =1 (1 ~ XtS k ) A 



Theorem 4.9. As formal power series, 

N 

S N (Xi,X 2 ,X) J[o,x\N J[ Ilfel^ 1 - x * s k) 

2 SF$ X \-N, -N + l- (1/A)(1 + Ai); -2N + 2- (1/A)(2 + Ai + A 2 ); t, s). 

4.3. Deformed holonomic system with finitely many variables. As pointed out in remark WM we 
already know that 2<SFi(a, 6; c; t, s) is a solution of piSCiF = abF for all 1 < i < n + m. It thus remains 
to prove that it is the only one that can be expanded as ^ K A k SPk (t, s) and such that it is equal to 1 
when (t,s) = (0,...,0). 

We first have to consider the binomial formula for super Jack polynomials. Let u be an indeterminate 
and define a F-algebra homomorphism e u : A — > ¥[u] by setting 

6u(Pr) = u, r G N. 

Stanley [H] (see also Section VI. 10 in Macdonald [53]) has shown that the corresponding specialisation of 
Jack symmetric functions is given by 

c.f. p.lOp and (|3.13p . When u = n, the above formula gives the evaluation Pi a \l n ). Analogously, for the 
super case, we define the algebra homomorphism e u : SA — > ¥[u] such that 

e u (Pr,a) =u, re N. 
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In particular, if u is replaced by n — am we have 

En-oro (Pr,a) = H - am = p r , a (l n+m ) 

and hence for any / £ SA 

e n - am (f) = /(l" +m ). 

In the last equation y(i ra + m ) means that f(x,y) is evaluated at x\ = ■■■ = x n = l,x n+ i = • • • = and 

Vl = ■■■ = Vm = 1, J/m+l = • • ■ = 0. 

We recall that SPk^ = <p(Pn ) where ip : A — > SA is the algebra isomorphism defined in ()3.6|) . Hence, 

Z*(SPM)= IT !j+^4^M , (4.24) 

In particular, for n,m £ No, 

r d aa K {%,j) + l K (i,j) + 1 

We here emphasize that SPji (l n+m ) might be zero, which is different from the Jack polynomials. For 
instance, if n = m and a = 1, then 

SpW(1™+«) = for any |k| > 0. 

In the finite variable case, Lassalle [28] defined generalized binomial coefficients (") by the following 
series expansion in the algebra of symmetric polynomials: 

Pn a \xi + 1, . . . ,x n + 1) ( K \ p a a \ x i-, ■■■,Xn) , 4 2g ^ 

This was recently lifted to the algebra of symmetric functions (i.e., infinitely many variables) |12j . The idea 
was to introduce a new indeterminate, called po, and then define the homomorphism # 7 : A F ( po ) — > A F ( po ) 
for any 7 £ F by setting 

^(p r ) = Er m (1p 3 , r>i. 

j=0 

Proposition 4.10 (Proposition 2.1, [H]). For any partition k, we have 

e x {pi a) ) _ ^ (k\ pi Q) 



E 



In a similar manner, we define the algebra homomorphism # 7 : SA$t po \ — > SA F ( po \ such that 

3=0 

By identifying po,a and ¥>(r>)j we get the following commutative diagram: 

A F ( Po ) — ej —^ A F(po ) 



^F(po, Q ) 1 — ^ ^F(po,c) 

It immediately follows from (|4.23j) , (|4.24|) and Proposition 14.101 that 

0i(Sp( a) ) = y, /k\ SP^ a) 
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Thus, we have the binomial formula for super Jack polynomials: 
SPto { Xl + l,...,x n + l; yi + l,..., y m + 1) = 

E ( K ) TflTTl sp i a) (^- • • , *»; yi, - , y m ). (4.27) 

where 

SPj?\\ n+m ) _ -j-r n-am + a(j - 1) - (i - 1) 
SPi Q) (l"+™) "~ (iJ ) £ «_ CT <xa K {i,j) + l K (i,j) + 1 

By summing up the (n + m) differential equations in the deformed system (|1.13|) and (|1.14[) , we get a 
single differential equation that preserves the space of super symmetric polynomials: 

C F — 

In the last equation, 

n+m , ^ s 

C n .m ■= E SCi - ( n - a>m)ab = D x n m - D 2 n m + ( c- -(n - am - 1) J E° TO 

i=l \ a / 

- ^a + 6+l--(n-am-l)j i?^ , m - (n - am)ab, (4.28) 
t=i j=i J 



where 



and 



™' m ' ^ 1 dt? 3 d S y a ^ U - tj dU V ^ - s,- 9s, 

»=1 1 j=l ■> l<t^j<n 3 l<i^J<m 3 



^V-V- 1 f k dF 1 k dF\ 1 x fc 1 <9F „ . 

2 E E 7T— : + -.jt - fc ( 1 + ^)E ^ V- ( 4 - 3 °) 



- U — Sj V 2 9ii a J 9s, 7 a •f-f J 9s, 

Z— 1 J — l J X J / J — 1 J 

The action of these operators on super Jack polynomials can be calculated explicitly. We use the 
notation e„ i£ N, for the sequence defined by (ej)j = <5y , where Sij is the Kronecker delta. In addition, 
if k is a partition, 

We also use the standard convention that terms in summations containing ku\ or «W are considered only 
if they arc partitions. 

Proposition 4.11. We have 

W = E(4)^^^i, 

= \k\SP™, (4.31b) 
Di „M a) = E ( K )(^ 1 + - am - i)) ^M 1 ^ SPtl , (4.31c) 

YWiA a 7^(i»+™) w 

< m SP^ = e K (a)SP( a \ (4.31d) 



where 



e K (a) = | /tj — 1 H — (n — am — i) j . (4.32) 
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Proof. One either applies the homomorphism ip n , m on the formulas of Lemma 3.9 in |12j or proceeds as 
in the Jack polynomial case [2U|36] and uses the binomial formula for super Jack polynomials ()4.27|) . □ 

The following proposition readily follows from equations (|4.31a[) - (|4.31dl) . 

Proposition 4.12. The formal series 

^ opt") (f. „\ 

F(h,. ..,t n , Sl ,...,s m ) = Y^ A H - " .\' ' (4.33) 

satisfies the equation C„_ m F = if and only if the coefficients satisfy the recurrence relations 

} j y ^ j ^ c + Kj j [n — am + am — i + 1)A kM 



= ( e K (a) + (a + b+l ) \k\ + (n - am)ab\ A K . (4.34) 

Let p < n and q < m. Moreover, let F Ptq denote the function obtained from the function F above by 
reducing the number of variables: 

F Pjg (ti, ...,tp,si,...,s q ) : = F(t±, . . . , t p , 0, . . . , 0, s l5 . . . , s q , 0, . . . , 0) (4.35) 

. SP^ a \ti,...,t p ;si,...,s q ) 

To ensure the uniqueness of the solution of £„ jm F = from the recurrence relations ()4.34|) . we make use 
of the following criterion, which was first introduced by Kancko in the non-deformed case (see Corollary 
4.6 in [35]). 

Definition 4.13 (Kaneko's criterion). The formal series (|4.33j) is said to satisfy Kaneko's criterion if 
£p,oF Pt o = for every p < n, and C- n ,qF n ,q = for every q < m. 

Lemma 4.14. Assume that [c]k =/= for any partition n G H n m . If the formal series (|4.33|) satisfies the 
Kaneko's criterion and F(0, . . . , 0) = ; then 

F(h, ...,t n ,S!,..., s m ) = 0. 

Proof. Let H 7hm = H n , m {jH n , m , where H n>m = {k£ H n>m \ Kn+1 = 0} and H n>m = {k£ H n , m \K n+ i > 0}. 
Without loss of generality, we assume n > 1. We will determine A K by induction from A( ) = 0. Note that 
the coefficient of A K {i) on the left-hand side of (|4.34[) is not zero except for the factor n — am + ani 
which is denoted by g n ,m( K i)- 

We first claim that A K = for every n G H n ^ m . It is sufficient to prove that F Pt o = by induction on 
p. The case p = 1 immediately follows from (|4.34j) because 3i,o( K i) = 1 + ckki > 0. Assume F P) o = for 
p < n — 1, i.e. A„ = if £(cr) < n — 1. For A K with K n = 1, put a = kua or cr^"' = k. Substituting this a 
into (|4.34|) shows that A a ( n ) is a linear combination of A a and A a{ i),i < n because of g n ,o(o~n) — 1 ; which 
implies A K = A a ( n ) = with k„ = 1. The case where k„ > 1 can be checked by induction on n n since 
g n ,o( K n) = 1 + aK n > 0. 

Next, we deal with the A K for k G H n ^ m . Every k G H n ^ m can be rewritten uniquely in the form 
K = («i, . . . , K n ) U7r' where 7r' = (k„+i, . . . , fv„+j, . . .) with n„+i < m, i.e. 7r = (7Ti, . . . , 7r m ). We prove that 
-Pn,g — by induction on q. If g = nothing is done. When q = 1, for ft € H n ,m with re = (rei, . . . , re„, 1, 0), 
i.e. 7r = (1), put er = K(„+i) or cr(™+ 1 ) = re. Substituting this a into ()4.34|) shows that A a ( n +v, is a linear 
combination of A a with er G H n m since c/ n .i(cr„ +1 ) = —a < 0, which implies A K = A a ( n +i) = 0. The case 
where £(k) > n+ 1, i.e. 7Tx > 1, can be checked by induction on 7Ti. Assume F n>q = for q < m — 1, i.e. 
^4cr = if er G H n rn ^i. For A K with 7r TO = 1, put er = or cr'" +1 ' 1 = k. Substituting this er into (|4.34|) 

and noting that 5n,m (cr„ + i) = -a, one obtains A^cn+i) = 0. The general case follows by induction on n m 
since g n ,m{Ki) ^ for any i > n + 1. □ 
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Lemma 4.15. If F(t\, . . . , t n , si, . . . , s m ) is one solution of the finite (n + m)-system, then for all p < n 
and q < m, F p ^ q (ti, . . . ,t p , Si, . . . , s q ) is one solution of the finite {p + q)-system. 

Proof. Proceed by induction on p and q. For instance, if p = n — 1 and q = m, consider the first (n — In- 
equations of (| 1 . 1 3[) in the finite (n + m)-system. Suppose ti ^ 0, Sj ^ 0, i = 1, . . . , n — l,j = 1 , . . . , m, let 
t n — >• 0, then we are done. □ 

Lemma 14. f 51 implies that if F{t\, . . . ,t n , si, . . . , s m ) is one solution of the finite (n + m)-system, then it 
satisfies the Kaneko's criterion. Returning to Lemma l4.f 41 we deduce the following. 

Lemma 4.16. Assume that [c]« ^ for any partition k G H nm . If F(t\, . . . , t n , s lf . . . , s m ) is one 
solution of the finite (n + m)-system, and F(0, . . . , 0) = 0, then 

F(ti, . . . ,t n , si, . . . , s m ) = 0. 

Theorem 4.17. 2SF^ a \a, b; c; t\, . . . , t n , Si, • ■ • , s m ) is the unique solution of the summed-up equation 
£n,m.F = subject to the following conditions: 

(a) F is symmetric in t = (t±, . . . ,t n ) and s = (s±, . . . , s m ) separately and satisfies the cancellation 
condition (| 1 . . 

(b) F has a formal power series expansion at (t, s) = (0, 0) with F(0, 0) = 1, 

(c) F satisfies the Kaneko's criterion: C p flF p fl = for every p < n and Cn,qF n ^ q = for every q < m, 
where F p>q is given by (|4.35l) . 

Proof. It follows from Theorem 14 .71 or Remark 14.81 that 2SF[ a \a, b; c; ti, . . . , t n , s-y, . . . , s m ) is one solution 
of the finite (n + m)-system, certainly implying that it is one solution of the summed-up equation C n , m F = 
0. Uniqueness is an immediate consequence of Lemma 14.161 □ 

We recall that every super symmetric solution of the deformed (n + m)-system is also a solution of the 
equation C n . m F = 0. Comparing Theorems 14.71 and 14 . 1 71 we then arrive at the desired statement for the 
existence and uniqueness of the solution to the deformed system. 

Theorem 4.18. 2 *SF'| a ' ) (a, b; c;t%, . . . , t n , s±, . . . , s m ) is the unique solution of the finite deformed (n + m)- 
system subject to the following conditions: 

(a) F is symmetric in t = (ti, . . . ,t n ) and s = (si,...,s m ) separately and satisfies the cancellation 
condition (|1.16[) . 

(b) F has a formal power series expansion at (t, s) = (0, . . . , 0) with F(0, . . . , 0) = I. 

Remark 4.19. As will be stated in the next section, the series 2SF\(a,b; c;t, s), as a function of (t, s), 
converges in some neighborhood of the origin in C n+m . Thus, Theorem 11.21 directly follows from Theorem 
14.181 Similarly, Corollary 11.31 is a consequence Theorems 14.91 and 14.181 and the convergence of the super 
hypergeometric series in finitely many variables. 

Similarly, for the semi-infinite systems, i.e. t = (t\, . . . , t n , 0, . . .), s = (s±, S2, ■ • ■) or ,s = (t±, t2, ■ ■ ■), s = 
(si, . . . , s rn , 0, . . .), the unique solution can also be given. Let us consider t = (ti, . . . , t n , 0, . . .) and s = 
(si,s 2 , ...)• 

Theorem 4.20. 2SF^ a \a,b; c;t, s) is the unique solution of the deformed semi-infinite system given by 
(|1.13j) and (|1.14[) with n < oo and m = oo, subject to the following conditions: 

(a) F is symmetric in t = (ti, . . . ,t n ) and s = (si, S2, • • •) separately and satisfies the cancellation 
condition (|1.16[) . 

(b) F has a formal power series expansion at {t, s) ~ (0, . . . , 0) with F(0, . . . , 0) = 1. 

Proof. Theorem 14.71 says 2 SFl a) (a,b; c;t,s) is a solution of the deformed semi-infinite system. Every 
solution can be rearranged as a series of super Jack symmetric functions 
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It suffices to show that the coefficient 7„ must be equal to 



A K = 



For any partition a and given m > max{l, cr Il+ i}, putting Sj = for j > m, F(t\, . . . , t n , s%, . . . , s m ) = 
/Cfc^Lo S|«|=fc 7kC k (2/i, • • • , y n , 0, . . .) clearly satisfies the finite (n + m)-system. Since a £ H ntJn , Theorem 
14.181 implies that j a = A a , as expected. □ 



Let us end this section by a direct generalization of Theorem 14. 181 which comes from Theorem 14.41 and 
a simple modification of Theorem 14.171 

Theorem 4.21. Fix n,m G No and let ot\,a.2, /3q, Pi, Jo be complex numbers such that /3q — i 1 —^ ~ 
are never zero for any k € H n m . Then the deformed system 



d 2 F dF 
U(ai + aaU)-Qjr + (A) + PiU)-^- + 7qF 



1 t k (. ,dF . dF 

V t 7- \{ui+a 2 t i )—-{a l +a 2 t k ) — 

m s / dF 1 <9i^ \ 

V— ^— (( ai +a 2 t i )— + -(ai+ a 2 s fc )— = 0, 1 < « < n, (4.36) 
^[U- s k \ dti a ds k J 



1 d^F / 1 \ 

-Sj(ai + a 2 Sj)-^ + \J} + (/3i - (1 + ~) a2 ) s j)^~ + (-a)j F 



+ V ^— [-{a 1 +a 2 s j )— + {a 1 +a 2 t k ) — = 0, 1 < j < m, (4.37) 

has a unique solution F(t, s) subject to the following conditions: 

(a) F is symmetric in t = (ti, . . . ,t„) and s = (si,...,s m ) separately and satisfies the cancellation 
condition (|1.16[1 . 

(b) F has a formal power series expansion at (t, s) = (0, . . . , 0) with F(0, . . . , 0) = 1. 

5. Further properties of super hypergeometric series 

In this section, we establish some important properties of super hypergeometric series p SF q with p < 1 
and q < 2. 

5.1. Special cases. Let us return to Theorem 14 . 2 1 1 and consider special values for the parameters oti, a 2 , 
/?o , Pi , and 70 . We are concerned with the solutions of the deformed system (|4.36|1 and (|4.37[) of the form 



F(t,s) = J2 c KSP K (t;s) with F(0,...,0) = 1. 



We immediately see that i$Fj (a; c; t, s) is the unique solution for the deformed system when 

(ai,a 2 ,A),/3i,7o) = (1, 0, c, -1, -a). 
Similarly, 2 SF^ a) (a, b; t, s) and Q SF^ a) (c; t, s) are the unique solutions for the respective cases 

(cti,a 2 ,Po,Pi,lo) = (0, -1, 1, -a - 6 - 1,—ab) and (cti, a 2 , /3 , Pi, 70) = (1, 0, c, 0, -1). 
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The other simple deformed systems can be solved explicitly. Indeed, one easily finds that the unique 
solution for the system with 

(ai,a 2 , A),^i,7o) = (0,0, 1,-1, -a) and (an, a 2 , /3 , Pi, 7o) = (0,0,1,0,-1) 

are respectively given by 

n m 

iSF^ a \a; t, s) = - UT a I^ 1 *i) ao , 
i=i j=i 

and 

n m 

»=i j=i 

5.2. Transformations. As pointed out in (T2], properties of the super Jack polynomials and generalized 
coefficients imply that the super series p SF q enjoy a duality symmetry that has no equivalent in the usual 
case. This duality can also be understood as follows. By interchanging (| 1 . 1 3|) and (|1.14[) . we can rewrite 
the deformed system in the same form but with a replaced by 1/a, and a, b, c replaced by — aa, —ab, —ac 
respectively. The uniqueness then implies the following duality relation. 

Proposition 5.1. We have 

2SF^ a \a, b; c; t, s) = 2SF^ 1 ^ a ^ (—aa, —ab; —ac; s,t), iSFq (a; t, s) — iSF (— aa; s, t), 

iSF^ a \a;c;t,s) = iSF^ a \—aa;—ac;—as,—at), qSFq (t,s) = o-SF (— as, —at), 

SF^\c;t,s) = n SF[ 1/a) (-ac;a 2 s,a 2 t). 

The next properties generalize some results obtained by Yan |46j in the usual Jack polynomial case. For 
convenience, we use the following convention b + as = (b + as\, b + as2, ■ ■ •) and = ( 1 t 1 Sl > i- S2 ; • ■ ■) f° r 
the complex number a, b and the sequence s = (si, S2, . . .). 

Proposition 5.2. We have the generalized Pfaff-Euler relations 
2 SF ( 1 a) (a,b;c;t,s) 



oo oo 

= IE 1 - ^ - Sjr a 2SFf\a, c-b;c; --— -- 
i=i j=i 

OO OO 

IR 1 - ^ - ^) ab 2SF^(c - a, b; c; -—, ~ 



i=l 3 = 1 



JJ(i _ t-y-a-b JJ (1 Sj )—(-»-b) 2 SF^(c -a,c-b;c; t, s), (5.1) 

i=l j=l 



and the generalized Kummer transformations 



oo oo 



1 SF { 1 a) (a; c; t, s) = J| e 4 * [J e" a ^ rSi^c - a; c; -t, -s). (5.2) 

i=i j=l 



Proof. Check directly that both sides of (|5.1[) satisfy the same deformed system and then make use of the 
uniqueness of Theorem 14.71 (|5.2[) follows from the third equality of (|5.1|) by setting t — t/b,s — s/b and 
letting b — > oo. □ 
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5.3. Convergence and integral representation. Up to now, the series were considered formally. The 
next proposition shows that the series in SP K (t;s) can be also treated as complex- valued functions on 
C n+m . The convergence of the super hypcrgcometric series actually follows the classical pattern. The 
proof will be given in Appcndix|Bj Here we emphasize that the part (3) of Proposition l5.3l below is slightly 
different from the usual Jack polynomial case (see part (3) of Proposition 1 [21], which claims that the 
series diverges at every nonzero point unless it terminates). Let us illustrate this point as follows: setting 
a = l,n = m = 1, we have SP^ (ti, s\) = on the line ti = si for any k ^ (0), thus p SFq(ti, s±) = 1 on 
the line t\ = Si, which is not divergent even if p > q + 1. 

Proposition 5.3. Let p SFq be the series in (|3.14[) with (t,s) G C n+ "' and a > 0. Let also r\ = 
max{a, a -1 } and ||(i, s) \\ = max-fl^l, . . . , \t n \, \si , . . . , s,„|}. 

(1) If p < q, then the series p SFq a ^ converges absolutely for all (t,s) £ 

(2) If p = q + 1, then p SFq°^ converges absolutely for s)\\ < l/(r\{n + rim)) . 

(3) If p > q + 1, then there does not exist a positive constant p such that pSF^ converges absolutely 
for ||(i, s)|| < p, unless it terminates. 

As mentioned earlier, the series V +\SF^ with p = 0, 1 are particularly important for us. Proposition l5.3l 
guarantees the convergence of such series in some neighborhood of the origin in C n+m . In some instances 
however, the domain of convergence can be extended. 

As an example, we first look at the binomial expansion of the deformed Cauchy product riij(l — 
XiWj)~ Pl ' l a . Wc know from Proposition 3.1 in [TH] that 



where 



1 h[f ] 1 , 1 



a \ K \ h ( a ) h ( a ') a 



and a = — . (5.4) 



By using Stanley's evaluation formula for Jack polynomials (see eq. (10.20) of Chapter VI in [29]), which 
can be written as 

hi 

we arrive at the conclusion that 



pi->(i«) = - ■ < 5 - 5 > 



B{^('»-^/--^' Woi ' iM) (5 - 6) 

Proceeding like in Appcndix[B] one easily checks that the right-hand side of (|5.3[) converges for all (x, s, t) S 
£l+m+n jugjj ^ na ^ \ Xi Sj\ < 1 and \xitk\ < 1, and so does i<S7q in the same domain. As a consequence 
we also have ^ 

which now converges in a larger domain than that given in Proposition 15. 31 

We now provide another proof that the deformed Selberg integral Sjv(Ai, A2, A; t; s) defined in (|1.10[) is 
in fact given by a 2SF1 series. 

Theorem 5.4. Suppose Ai,A2 > — 1, A > 0, \sj\ < 1 and \tk\ < 1- Then 

S N {X U A 2 , A; t; s) = S N (X 1 ,X 2 , A) 2 SF^ X) (-N, —N + 1 — (Ai + 1)/A, -2N + 2 - (Ai + A 2 + 2)/A; i, s) . 
Proof. First set a = 1/A and let 9 denote the set of points (x,s,t) in £ N + m + n such that < Xi < 1, 
\sj\ < 1 and \tk\ < 1. According to (|5.6p . the series iSJ^q (A/a; x; i, s) converges to Il^liIIj=:i(l ~ 
a; i s j)rife=i(l ~ Xitk) -1 / 01 everywhere in the domain 0. The latter function is continuous and bounded 
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on the same domain. Application of Lebesgue's dominated convergence theorem in Proposition 13.41 then 
proves that 

SWAi, A a> 1/a; a; t) = SW(Ai, A 2 , A) 2 5P 1 (a!) f — — — + Ai + 1, 2N ~ 2 + \ 1 + \ 2 + 2- t, s 

\ a a a 

The interchange of s and t together with the use of Proposition 15.11 finally establishes the theorem. □ 

5.4. Super Jacobi polynomials. The super Jacobi polynomials were introduced in [32] ■ They can be 
defined the unique super polynomials &/« (t, s;a) satisfying the two following conditions |12j : 

(1) = SPW + £ c^SPl a) (5-8) 

(2) « m Dl m + ( 7 + l)El rn - ( 7 + n + 2)E^ m ) SJ^ = e{ SJ™ (5.9) 
where each coefficient c KM depends upon a, 7, 77 and po — n — am. The eigenvalue is explicitly known: 

4 = ~J2 Ki («t-l + |(P0-0) -(7 + *? + 2)|«|. (5.10) 

Now, we have established in the last section that 2<SFi(a, &, c; t, s) is a solution of the form J2 K (I k SPk (t, s ) 
to the following equation: 

Di m - Dim +( c - ^-^l <m -(a + b+l- ^zl) E i m ) F = abp F. (5.11) 



Moreover, according to Lemma 13.21 2<SFi (a, b, c; t, s) is a polynomial if 

M 

a = —N, and = — . 

a 

With this choice of parameters, the highest term of 2SF1 (a, b, c; t, s) is given by the largest possible partition 
in the fat hook H n ^ m whose fist part and length are respectively equal to N and M, which is 

K max = (N n ,m M - n ). (5.12) 

Moreover, one easily verifies that e^ ma x = abpo whenever 1 + 7 + 77 = + 6— (po — l)/a, a = — TV, and 
b = M/a. Collecting all the preceding observations, we conclude the following. 

Proposition 5.5. Let n max as above, 77 = —7 — 1 + m — N + (AI + 1 — n)/a and c = 7 + 1 + (7^0 — l)/ct. 
Then the super Jacobi polynomial Sjj^L2) (t, s; a) is proportional to 2<SFi (— N, M/a, c; t, s). 

In principle, 2SF1 (— N, M/a, c; t, s) cannot be represented as a deformed Selberg integral since it 
would correspond to the case Ai = —A/ — aN + a — 1 which is negative. However, as we pointed out at 
the end of Section^ the multiple contour integral Tjv(Ai, A 2 , A; t, s) is well defined for such values. 

Corollary 5.6. Let A G Z +7 Ai = —M - 1 - X(N - 1), A 2 = -\{N -m + j)+M-n-l>-l and 
i] = —7 — 1 + m — N + (M + 1 — n)/A. Then, the super Jacobi polynomial Sjj^Zl) (t, s; A) is proportional to 



r (o+) jv n „ , _ r .f.\ 

I nM+i*.**,.*, 



1 Ji fjiiik=i( i - x j s ky 

6. Applications to /3-Ensembles of random matrices 

In this section we apply our results on super hypergeommetric series to the classical /3-Ensembles 
of Random Matrix Theory, namely the Jacobi, Laguerre (Chiral), Hermite (Gaussian) and Circular /3- 
Ensembles. Note that in following paragraphs, the indeterminates t = (ti, . . . , t n ) and s = (si, . . . , s rn ) are 
always considered as finite sets of complex variables. Unless otherwise stated, the series considered below 
are convergent. 
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6.1. Jacobi /3-ensemble. The joint probability density function of the Jacobi /3- Ensemble (J/3E) is equal 
to 

1 N 

— — — — n x ^(i- Xi ^ n \ Xj - Xk f. (6.i) 

S N {X u X 2 ^/2)fJ { i^<t<jv 

According to Corollary 11.31 (or equivalently Theorems 14.91 or I5.4|) . the average of ratios of characteristic 
polynomials over this ensemble is given by a particular 2 SF\ series. 

Proposition 6.1. Whenever \ti\ < 1 and \sj\ < 1, 

I N n m \ 

(na-^na-^ 2 )) = 

' J/3E 



\k=l i=l j=l 

2 SF[ P,2 \-N, -N + l- (2//3)(l + Ai); -2N + 2- {2//3)(2 + X 1 + A 2 );i; s). (6.2) 



Moreover in the case where |fj| > 1 and \sj\ > 1, following Lemma T2.3[ we have 

/ N n m \ n tci 

( n ( n>* - *•) - **)-*'*) ) = i-ir-- m]N n $ n *;* 

J/3E 



3 

\k=l i=l j = l I lfjF i=l j = l 



■ 2 SF^ /2 \-N, -N + l- (2//?)(l + Ai); -2N + 2- (2//3)(2 + Ai + A 2 ); -js J), (6.3) 

which satisfies the finite (n + m)-system with a = f3/2,a = —N,b = (2 //?) (1 + Ai + X 2 + po) + N — 1, c = 
(2//3)(Ai +po) and Pa = n - (j3/2)m. 

It is tempting to conclude from the last observation that the expectation given on the left-hand side of 
(|6.3[) is equal to 2 SF^^ 2 \a 1 b;c;t,s), with a,b,c equal to the values just given. However, the uniqueness 
theorem does not apply for this case since the expectation is not analytic in (i, s) = (0 . . . , 0). In order to 
extend the range of possible values for t and s, we introduce another type of correlation function: 

where it is assumed that Ai g" Z, A2 E N, j3 E 2N, and ^ stands for the contour introduced for the definition 
of the function K% in (|2.24j) . Moreover, 

Z N := (e 2 ^ -1) N S N (X' 1 ,X 2 ,X) = 

, ^N, n^rr 1 r(i + A + jA)r(i + A 2 +jA) 

{2m) M) l[ r(l + A)r(2 + Ar + A 2 + (N + j - 1)X)T(-X[ - jX) (6 ' 5) 

where A = /3/2 and A^ = X\ + n — Am. Note that the normalization constant Zjy has been evaluated 
thanks to the identity (s'mirz)/ir = l/(r(z)r(l — z)); it is such that the right-hand side of (|6.4p equals 1 
when Sj = and U = for all This allows us to apply the Theorem 11.21 to Corollary 12.61 and obtain 
the exact expression for the correlation function. 

Proposition 6.2. Let X±, A 2 , as X as described above. Then for all ti,Sj E C such that |^| < 1 and 
\ Sj \ < 1, 

/n nr=i(^-*i) \ 
\Mn;: 1 (-- J )^/^ E 

= aSFf 72 ' (— iV, JV - m - 1 + 2(Ai + A 2 + n + 1)1(3, 2(n + A x )//3 - m; i; s) (6.6) 
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We know from Lemma [3721 that the last series terminates if for some positive integer M, 

2 2M 
N -m.-l + -(X 1 +X 2 + n + l) = — . (6.7) 

P P 

In such a case, the series also becomes a rational function in (3, well defined for all j3 > 0. By analytic 
continuation, the restriction (3 = 2N can then be removed from the definition of the correlation (|6.4[) . 
Moreover, according to our discussion in Section IST41 we also conclude that when the condition (|6.7j) is 

satisfied, the correlation function is proportional to a super Jacobi polynomial SJ^2l(t, s; (3/2), where 
7 = 2(Ai + l)//3 - 1 and r? = 2(A 2 + l)//3 - 1. 

As immediate consequences of formulas (|6.2j) and (|6.6[) , we compute the following scaled limits at the 
left hard edge of the spectrum (see [14] for more details on the spectrum edges in Random Matrix Theory). 

Corollary 6.3. With the appropriate choice of parameters X\, A 2 and (3, we have 

^ ( II (IK 1 + na + 2 ^r m ) = IP* II *- {m)Si - 

\k=l i=l 3 = 1 I 1=1 j=l 



and 



t 



iToo ( n (n^ - ^) no* - ^- m ) ) = ^ ,2) + ^ - -) • 

U=l »=1 3=1 / ^j^g 

6.2. Laguerre /3-Ensemble. The Laguerre /3-Ensemble (L/3E) is characterized by the following joint 
probability density function: 

-J—II^e-**/ 8 II I**"**!'- (6-8) 

i=l l<J<k<N 

The normalization constant is easily derived from the Selberg integral [T4"ll32j : 

W - (h(i+x,)N + pN(N-i)/2 jt r(l + ^/2 + ^/2)r(l + A 1+ j/3/2) 

^^'^ l /? 11 r(i + /3/2) 

The density given in (|6.8[) can be obtained from the J/3E density (|6.1[) by making the following limiting 
change of variables: 

Xt i — ► Xi/L, A 2 i — > (3L/2 7 L\ — > oo. (6.9) 
By applying the same limit transformation to the correlation f|6 .4[) and assuming that the variables Sj and 
tj transform as Xj, we get the following formula involving which now denotes a Hankel type contour 
such that each variable x^ starts at oo + i0 + , then encircles all the variables Sj in the positive direction 
without crossing the interval [0, oo), and ends at the point oo + i0~. 

Proposition 6.4. We have 



1 \.lj = l\ X k s j) ' 



where the normalization is chosen so that the right-hand side of (|6.10[) is equal to unity when U = and 
sj = for all 

The scaled limit at the hard edge is easily computed. 
Corollary 6.5. We have 



iV- 
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Now, when applying the same transformation (|6.9j) to the expectation values given in equations (|6.2|) 
and f|6 . 3[) , one formally gets 



I N n m 

(ndi^-^iK 1 -^)"" 72 

U=l 1=1 3=1 / L/JE 

2 5F (/3/2) (-^, -iV + 1 - (2//3)(l + AO; -t, -s), (6.11) 

and 

I N n m \ n m 

( n ( nc** - **) no* - ^)-^ /2 ) ) = {-ir-^ N n n *r N 

\k=l i=l j = l I L/3E i=l j=l 

2 SfW*\-N, -N+l- (2//3)(l + Ai); ~, -i). (6.12) 

As we already pointed out, if a and b are not equal respectively equal to — N and M/a for some N, M S No, 
the power series for 2$Fq (a, t, s) is in general divergent (see Proposition l5.3l part (3)). Thus, apparently, 
formulas (|6.1ip and (|6 . 1 2[) do not make sense. However, as will be shown below, there is an integral 
representation for 2$Po that allows us to circumvent the convergence difficulty. 

Let us first consider the expectation of a Jack polynomial in the L/3E, which can be computed by making 
use of formula (|3.17l) : 

(Pjr\x) ) aw = P^(l N ) [A x + 1 + (N l)/at\ (6.13) 

\ / LpE 

On the right-hand side, we make the change of variables Xi = uyt, where < u < oo and 53 j yi = 1, which 
leads to 

Ca I P^(y)l[yt n 1% ~ y*\ Vad °N = r , r( !°! ^ Pl a) (i N )[\i + 1 + (N- l)/a]W. (6.14) 

In this equation, Ajv denotes the set of points in the hyperplane 53i Vi = 1 with y^ > 0, dajy stands for 
the Lebesgue measure on Ajv such that the left-hand side of (|6.14[) equals to one when k = (0), while 
C A = a c °r(c )/iy Al , 2/Q , N and c = AiiV + 7V(7V - I) J a + N. 

Following the method exposed in [6], which is concerned with the Jack polynomial case, we now define 
a new type of super series 

2 SPr (a, ft; c; t; s) = V V Wk Mk SP™ (t, s), a, b, c e C, c + 0, -1, -2, . . . . (6.15) 
Then, proceeding as for Propositions 13.41 or 17.21 and using (|6.14p . we get 

N nn /-, , \ N 

Ai TT i i2/n i 



c - f n nfl¥^gfe n^ n ^ - «*i a/B ^. 

Ja n k=1 ll i= iU ^fcSjj . =1 



2 sM 1/a \-N, -N + 1 - a(l + Ai); A X 7V + 7V(7V - 1)/q + TV; -it, — s). (6.16) 

Furthermore, 

( ft (na - u*>) fia - «^)- /,/a ) ) =fk)f du 

\k=l i=l j = l I L/3E 

u c °-^" 2 SFf /2) (-iV, -N+l- (/3/2)(l + Ai); c ; -ut, -us), (6.17) 
where c = AiiV + (2/0)N(N -l)+N. 
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At this point, it should be stressed that the right-hand side of (|6.17|) is formally equal to that of (|6.11|) . 
Indeed, if one expands 2 SF^^ 2 ^ and then applies the integral transform J Q °° duu c °e~ u term by term , one 
recovers the formal power series defining 2 SF^^ 2 \—N, —N + 1 — (/3/2)(l + Ai); —t, — s). This suggests a 
new definition for 2 SF : 

2 SF (Q) (a,&;i; S ) := — - / u^e'^SF^ (a,b; c ;t; s)du, (6.18) 

1 (Co) Jo 

The integral is well-defined at least when tj > 0, Sj > for alH, j, a= —N and b = — N + 1 — (/3/2)(l+Ai). 
Our new definition thus resolves the convergence problem. In what follows, the notation 2 SF is used in 
the sense of (|6.18l) . 

Notice that the function 2 SF^ (o, b; c; t\ s) shares some properties with the standard super hypergeo- 
metric functions 2 SF^ (a, b; c; t; s). 

Proposition 6.6. (i) The power series defining 2 SF^ a \a,b] c;t, s) converges in the polydisk < 1 , | < 
l,i = l,...,n;j = l,...,m}. 

(ii) We have the duality relation 

2 SF^ a \a,b;c- 1 t,s) = 2 SF^ /a) (-aa, -ab; c; -s/a, -t/a). (6.19) 
Proof, (i) One compares the series 2<SFx ( a ' ^> c ' s ) w ith the series 



1 ^ ) (a;i, S ) = ^^l^ r 5PW( i)S ). 

By virtue of the formula 



fc=0 |«|=fc 



i5F w (a; t, s) = uy a J^ 1 

i=i j=i 

the series on the left-hand side converges in the polydisk in question. Now, given that the ratio [£>]« /(c)\ K \ 
has at most polynomial growth in the former series also converges in the same polydisk. 

(ii) One does the same calculation as in the proof of Proposition 7.2 in [12]. □ 

We return to our main subject. The above computations allow to give the following exact expressions 
for the averages of ratios of characteristic polynomials in L/3E. 

Proposition 6.7. Let 2 SF denote the function defined by (|6.18[) . Then, 

N 

13/2 



\k=l i=l j=l 



L/3E 



2 SF {P/2) (-N, -N + l- (2//3)(l + Ax); -t; -a), (6.20) 



and 

IN n 



n ( nc** - *•) ir** - s ^~ m ) ) = m)iv n $ n 

\k=l i=l j=l / L/3E i=l j=l 



2 SF a W2 \-N, -N + l- (2//3)(l + Ai); ~;~). (6-21) 



Corollary 6.8. We have 

N 



n ( na + + l^ 72 ) ) = n et< n e ~ (/V2)s 
\fc=i i=i j=i I L/3E i=i j=i 
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Proof. Combining (|6TPT)l and (|O0j) . we find 

3 SF<F /2 \-N, -N + l- (2//?)(l + Ax); ^, ^) 



(2j !7? C ° /°° duu c °- 1 e-P Na 'ft" !l SFi /i/ *\-N, -N + l- (/3/2)(l + Ai); c ; (2/j8)ut, (2/,8)ua). 
1 ( c oJ Jo 



SFf /2 >(-iV, -TV + 1 - ( / 8/2)(l + Ax); c ; (2/0)ut, (2//3)us) — ► SF (,3/2) K us), 



Since, as TV — > oo, 

2 

and 

r ( c o) 

we get the desired result from the fact that SF^ /2) (t, s) = Jl-Li e U YYJli e" (/j/2)Sj . □ 

The second integral in Proposition 16. 71 can be used to give expressions for some spacing distributions at 
the hard edge of the spectrum. Now, by definition (see [T3] for more details) 

1 r N 

E N j 3 (0- : (0, S y,x a e-^ 2 )^- / n< e ~ fe/2 \x j -x k fd N x 

W a ,p,N 7 [s ,oo)« tl l<j<k<N 

e -N0s/2 p 

Wa,/3,N J[0,^)«" l<f<k~<N 

where the second equality follows from the change of variables x% i— \ Xi + s. From this, the distribution of 
the smallest eigenvalue is given by 

d 



W a ,p,N J[s,ao) N 
N 

/ l{ Xi + S ) a e-^' 2 \ Xj -x k fd N x, (6.22) 



p { P(0; s; a) = ~E N , P {0; (0, s); x a e~^ 2 ) 



„ N-l 

/ ]x^x l + S re-^ 2 \x 3 -x k fd N ~ l x, (6.23) 



Wa,f),N J[Q^)N-i A A l< j<k <N-l 

where the second equality follows from the differentiation of the first equality in (|6.22p and then from the 
change of variables 

For a € No, the final integrals in both (|6.22[) and (|6.23[) can be expressed as (non super) hypergeometric 
functions [Ml Proposition 13.2.6]. With the aid of integral (|6.12[) they can also be written in terms of the 
super hypergeometric functions, but this time, for more general values of a . 

Proposition 6.9. For a = n — (/3/2)m > —1 with n,m 6 No, we have 

E N , (O;(O,s);x a e-^ 2 ) = e -^l s (--^ 2 SF^^ f—N, -N + 1 - 2//3; (1)* 

and 

p W(0; S ; a) = ^hML Ne -N^/2 s (n^ m) N 2SF W2) (_ „ + _ N 2//? . 

p W a ,p,N \ \sJ \sJ 

6.3. Gaussian /3-Ensemble. The Gaussian /3-Enscmble (G/3E) refers to a set of real random variables 
x = (xi, . . . , xn) whose joint probability density function is given 

1 N 

i_JJ e -^/ 2 ft \xj-x k \ p , (6.24) 



where the normalization [ 14 32 



Gr n 

P ' JV i=l l<j<k<N 



N-l 



n o-N/2-l3N{N-l)/A (n \N/2 TT ^(1 + 0/2 + jfi/2) 

Gp,n = fi (2tt) 11 r(l + /3/2) 
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From our perspective, the expectation values of ratios of products of characteristic polynomials in the 
Gaussian /3-Ensemble seem more difficult to tackle. We nevertheless have the following. 

Proposition 6.10. 

I N n rn \ 

n(n(xk-U)l[{x k -s 3 )-P/ 2 U =In,m,N (6.25) 



\k=l i=l j = l 



G/9E 



whe 



limJ-Lr N 2 SF^ (-N, |(1 + Po ) + 2L 2 ; |p + L 2 ; 1(1 - 1), 1(1-1) 



and p =n — (/3/2)m. 
Proof. In ()6.3|) . we set 



a*->i(l-^), 4i ^i(l-|), A 1 = A 2 = ^L 2 /2 (6.26) 

and take the limit L — > oo. Note that under the above change of variables the density in the J/3E becomes 
the density in the G/3E. Finally, the properties of the Gamma function imply 

u S N ((3L 2 /2+ Po ,(3L 2 /2,(3/2) = 2 _ pgN 
l^oo S N {(iL 2 /2,(3L 2 /2,l3/2) 

and the result follows. □ 



One easily verifies that I n . mi N does not reduce to a single super hypergeometric series. Nonetheless, it 
satisfies a deformed holonomic system of differential equations, which is obtained by substituting (|6.26[) 
into (PT5j) and (fTHp . but with a = [3/2, a = -N, b= (2/0) (1 + A x + A 2 +p ) + N- 1, c= (2/ /3){X 1 +p ). 

Proposition 6.11. Let a = 13/2, a = —N. and b = 2. Then, I n ,m,N satisfies the deformed system 




(6.27) 



1 8 2 F ^ dF F y, _L / dF dF 

a ds 2 3} d S] a a k J-^^ Sj - s k \d Sj ds k 



+ E^-f-^ + f : )=0,. ? = l,..,m, (6.28) 
^ Sj -t k \a dsj dt k I 

and the cancellation property 



(dF 1 dF" 

\ dU a dsj y t 



(6.29) 



Remark 6.12. By interchanging (|6.27l) and (|6.28p . we can rewrite the deformed system in the same form, 
except that a, a, and b are respectively replaced by 1/a, —aa, and —ab. This shows that the solution of 
the system still enjoys a duality symmetry. 
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6.4. Circular /3-Ensemble. By making use of the general identity (see Eq.(1.16)-(1.18) in [15] or [14] 
for more details about this argument) 

(xi ■ ■ ■x N ) c ~ 1 f(xi, . . .,x N )d N x 

[o,i] N 

e i «>i+"+ e »Kf(-e ie \. . . , -e WN )d N e, (6.30) 



(2 sin O 



N 



valid for / a Laurent polynomial and Rc(£) large enough so that the left-hand side exists, the Sclbcrg 
integral formula (|1.3I) can be transformed into its trigonometric form 

S N (\ lt A 2 , A) = (_i)^(Af-DA/2 . 1 M N (a, 6, A), (6.31) 

[Z sin TTu) 

where Ai := —b — (N — 1)A — 1, A2 := a + b and the Morris integral 



M N (a,b,X):= [ TT e^ a ~ b ^ k |1 + e l9k \ a+b TT \e ie * - e Wk \ 2X d N 6 (6.32) 

J[-^M" k =i j<k 

ro~\N I ~f~f r(l + A + jA)r(l + q + b + jA) 

= ( } l[r(i + x)T(i + a + 3 x)r(i + b + jX) (6 ' 33) 

for a,b,XeC such that Rc(a + &+l)>0,A>- min{l/7V, Rc(a + b + 1)/(N - 1)}. 

The above formulas readily imply that Theorem 14 . 91 and formula (|6.2I) have a trigonometric counterpart. 

Proposition 6.13. 

N N 



i-_ / nn eKa " 6)9fe ii+^r h n( i +^ fe 

MAr(a, 6, 1/a) ./[-^jv \ aj. 11 jll 



fe=i fe=i p=i 



J(l + s g e 4efc )- 1/Q J] |e^ -e l9fc | 2/Q d Ar = 2 5F 1 (1/Q) (-iV,a6;-^+l-a(l + a);t;s). (6.34) 

9=1 / j<fc 

Now, the joint probability density function for the Circular /3-Enscmblc (C/3E) is equal to 

J_ JJ | e ^- e ^|^ (6.35) 
C/3,iV l<i<t<iV 

where the normalization constant is C 0tN = {2n) N T(l + Nf3/2)(T(1 + /3/2))~ N [HI32]. More generally, 
when a = b,t = and s = 0, the integrand on the left-hand side of (|6.34[) is real and referred to as defining 
the Circular Jacobi /3-Ensemblc (CJ/3E) Q~lj. The formula below directly follows from (|6.34l) . 

Proposition 6.14. 

N 



n(n( i +^)iK i +^)^ /2 



\k=i t=i 3=1 / CJ/3E 

2 SF^ /2) (-N, (2/(3)b; -N+l- (2//3)(l + 6); t, s). (6.36) 

7. A MORE GENERAL DEFORMED SELBERG INTEGRAL 

We end the article by explicitly computing a more general Selberg integral than 5jv(Ai, A2, A; t; s). The 
approach here differs from the previous sections in the sense that no deformed holonomic system for this 
integral is yet known and the series considered are kept at the formal level. 

Wc first need to recall and then generalize some concepts exposed in Section [3] Let x = (xi, . . . , x n ), 
y = (?/i, . . . , y m ), and 7 G F. The deformed power sums, which are defined as 

Pr,-y( x >y) = Pr( x )-lPr(y), r G N, 
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generate an algebra denoted by Af 7 (x,y). There is a natural homomorphism ipn}m ■ A(z) — > M~ l {x 1 y) 
given by 

fnJniPr) = Pr,y{x,y). 

We define the 7-generalized super Jack polynomial SPi a,1 \x,y) as the image of the Jack symmetric 
function pjf^ under tp^Jn, that is, 

SP^\x,y) = <p%l(P^). (7.1) 
We remark in particular that if 7 = ra for some r £ N, then 

^ 0^1 3 ■ • ■ i X n ) 2/1; * • * 5 2/n) '^"re 0^1 3 ■ • ■ 3 X n , %1 3 • • • 3 ^rn) 

whenever z x = • • • = z r = y x , . . ., z r („_ 1)+1 = • • • = z rn = y n . 

We actually know little about the 7-super Jack polynomials. For instance, we have not identified the 
kernel of (pn)m in the C-span of the all Jack symmetric functions P™. This does not cause any problem 
however for the formal definition of the 7-super hypcrgcomctric series. 

Definition 7.1. Fixp, q £ No and let a\, . . . , a p , b\, . . . , b g be complex numbers such that (i— l)/a— bj ^ No 
for all i £ No. We then define the hypergeometric series 

P SF^ ^(a 1 ,...,a p ;b 1 ,...,b q ;x,y):=}_^ ^ Tw^ 775) ■ ^ 

fc=0 | K |=fc [°iJk ' ' ' l"q\K UK 



The following proposition is a formal generalization of Theorems 14.91 and 15.41 
Proposition 7.2. As formal power series, 

1 r A n;ui-^) n rWV 

^(Ai.Aa.l/a) / [0>1] * H nr=i(l-^) 7 " Al ' A " 1/oiWd * 

= 2 SF^ 1/an) (-N, -N + 1 - a(l + Ai); -2JV + 2 - a(2 + Ai + A 2 ); t; s). 

Proof. First, we recall identity (5.4) in Chaper VI of [29] for the Jack symmetric functions: 

00 

n (i - x iZj ) = ^(-i)i K ip( Q )(x)Pi, i/tt) (z). 

i,i=l « 

Second, we act with <y9^rn on both sides with respect to z and then restrict the number of variables x by 
setting Xi — for all « > N. This yields the identity 

fc=l l-U=lv fc " k 
where now x = (x±, . . . , ojjv). Note that the left-hand side of (|7.3I) immediately follows from 

ft iferf^ - E ^(*)M*) - ( 7 - 4 ) 

Now, we know from formula (|3.17[) the value of the integration of a Jack polynomial Pk (x) with respect 
the Selberg density. But according to result (10.20) of Chapter VI in [29] . 

P(«)n^= TT N + a a' K (i,j)-l' K (i,j) _ a^[N/ai a) 
where the second equality follows from (|3.1ip — (|3.13p . Note that since 
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we see from (|7.5|) that Pk(1 n ) = whenever £(k) > N. Thus, the formula (|3.17j) holds true for any 
partition k. 

Finally, combining (|3.17[) . (|7.5[) and the property 

[o]W = (-a)-W[-ao]^, 
on can integrate term by term on the right-hand side of (|7.3I) and get the desired formula. □ 

Similar calculations lead to the following trigonometric version of the generalized integral. 
Proposition 7.3. As formal power series, 

1 



M N (a,b, I /a) 



X\{l + t p e^)\[{l + Sq e^)- 

\p=l 9=1 



N 



J e $<(«-6)fl*| 1 + e tf k |a+5 JJjgi*, _ e i6 k] 2/ a Q 
fe=l j<k J 

= 2SFi 1/a ' 7 \-N, ab; -N + 1 - a(l + a); t; s). 
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Appendix A. Proof of Lemma 14.21 

It is sufficient to prove the lemma in the case where i = 1. Suppose that Afc (r) / for some fco <E No, 
writing Afc (r) = Yliez^i r 1 then there exists Iq = . . . , i no , 0, . . .) such that Cj # 0. Taking n = 
max{£;o + 1) no} and putting yj = for any j > n, the equation (|4.10j) reduces to 

n-1 

Y / ^k(ri,...,r„,0,...)r[ 1) =0 (A.l) 

with Afc (ri, . . . , r n , 0, . . .) # 0, which leads a contradiction with the Lemma, p. 371, James |19j . The 
lemma, which plays a crucial role in the proof of uniqueness of solution of the system (|4.2|) . was first stated 
by James but its proof was not given. Next, we give one proof by induction. 
Recall the James' lemma says that 

n-1 

X>fc(n,--.,r„)r<%2,...,y„) = (A.2) 

k=0 

implies X (ri, . . . ,r n ) = 0, Ai(ri, . . . ,r„) = 0, . . . , A„_i(ri, . . . ,r n ) = 0. 

For n = 1, clearly Xo(ri) = 0. Assume that we have completed the proof when n < m. We consider the 
case where n ~ m. 

Writing A fe (ri, . . . , r m ) = J2iez m Cj'r 1 ^ = 0, 1, . . . , m — 1, we claim: for I = (i ls . . . ,i m ), if i m = 0, 

then = 0, k = 0, 1, . . . , m — 1. In fact, When i m = and < k < rn — 1, putting y m = 0, (|A.2[) 
reduces to 

m-2 

£ A fc (ri m \ . . . , ^.O^^fe, . . . , y m _ l5 0) = 0. (A.3) 

fe=0 

By induction, we have Afc(rJ m ' 1 , . . . , r^^i, 0) = 0, implying that cj fc ' =0, 0<fc<m — 1. When i m = 
and k = m — 1, putting yi = 0, since each nonzero term of \k{fi, ■ ■ ■ , r m ) (fc < m — 1) contains the factor 
r m , it must follow that A m _i(rJ 1 ' 1 , . . . , r^_ ly 0)r^:_i = 0; which results in cj™ 1-1 -* = 0. 
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For the coefficient Cj with i m > 0, if {Cj 7^ : I 6 £ m , i m > 0,k = 0, . . . , m — 1} is not empty, 
denoting by Cj k °^ one nonzero coefficient such that the last component i m of Iq = (ii,...,i m ) is the 
smallest of {i rn : Cj =/= 0, / G £ m , k = 0, . . . , m — 1}. Rewrite the equation (|A.2[) as 

Afc(ri,...,r m ) (i) 

2^ tz r k (v2,---,y m ) =o, (A.4) 

using the above claim, we obtain cf o) = 0, which is a contradiction. 

Appendix B. Convergence of super hypergeometric series 

We study the convergence of the series given by (|3.f4[) . We assume that a > and x = (xi, . . . , x n ), y = 
(j/i, . . . , y m ). An estimation of SPk (x, y) is first given by the following lemma. 

Lemma B.l. Let \\(x,y)\\ = max{|xi|, . . . , \x n \, \y±\, . . . , \y m \} and ri = max{a, 1/a}. There exists a 
positive constant C„ jm depending only on n, m such that 



\SP<?\x, y)\ < C n , m ^hi a) /h' ( :\ ri {n + r im )\\(x,y) 

where 

h' K = [a K (i,j) + -l K (i,j) + - 

Proof. Put \n\ = k > 0, and write T = ||(x,y)|| and 

If we apply Cauchy's inequality to the sum on the right-hand side we obtain 

|(7|=fc |<x|=fc 

where z CT = (l tTl 2 ,T2 • • • )a 1 \a 2 \ 

On the other hand, by definition of super Jack polynomials we have 

P^(Z U ...,)= J2 XaPa(Zl,... t ) 
\(r\—k 

and hence 

\<y\— k 

Here we use the scalar product of Jack functions, see (10.16), section IV, [2"5] . 
Since [29] 

— — = /ife := coefficcnt of u fc in TT(1 — Ziii)~ l , 

\<r\=k Za i>l 

after the action of (pn,m on the both sides we get 

i \ n m 

^ Pa '- a ^ y > = coefficcnt of u k in \{(\ - x t u)- 1 J](l - x 3 u)- a . 

\<r\=k Za i=l 3=1 

Hence, it follows from 

(a)d <rfand ^ + ^ < (d + l)fc d 



d! " 1 V k 
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that 



This gives 



E 

\a\=k 



< ( U + m ~ 1 + k ) (r lT ) k <(n + m)k^-\nTf. 

Za V K ) 

ST \PoA 2 w , . y^k ^ p a - a (\x\,\y\) 
E ^y^'-iln + wOT) ^ 

|<x|=fc 



CT I — k 



where (C*„, m ) 2 := sup fc {(?i + m)k n+m - l {n + mri)" fc }. □ 

Theorem B.2. (1) If p < q, then the series p.!4j) converges absolutely for all (x,y) <E C" +m . 
(2) If p = q + 1, then (|3.14[) converges absolutely for \\(x,y)\\ < l/(r 2 (n + rim)) . 

f5j i/p > g+ 1, i/ien there does not exist a positive constant p such that (|3.14[) converges absolutely for 
\\{x,y)\\ < p, unless it terminates. 

Proof. We compare the series (|3.14[) with the hypcrgcomctric series with one variable z 



p F q (a 1 ,... > a p ;b 1 ,...,b q ;z) = ^ - 



which has radius of convergence J2 = oo if j> < g, R = liip = q + 1, and i? = 0ifp><7 + l unless it 
terminates. 

Note first that every k € H n .m, can be rewritten uniquely in the form k = kqUt:' where kq — (k\, . . . , n n ) 
and 7r' = (k„_|_X) • • • j Kn+ii ■ ■ •) with K n+ i < m, i.e. 7r = (7Ti, . . . , 7r m ). Then we have 



a 



Write 

a„j = a u - (l/a)(i - l),b m = b v - (l/a)(i - 1). 



a,,,- = n — aa, 



a(j-l), 



b V j = n — ab v — a(j — 1), 1 < u < p, 1 < v < q, 1 < i < n, 1 < j < m, 
Set r 2 = minjo;, 1/a}, obviously we have = 1. Since 

hi a) h' { f = [] f 1 + m - J + - i) J f /«< -3 + ^*3-*+ 1) 
(i,i)e« ^ ' ^ 



(j',i)67r V 



it is easily verified that h^h y K ' > r^^ (ki\ ■ ■ ■ K n l) 2 (tti\ ■ ■ ■ 7r m !) 
By Lemma |B. 11 we have 



EE 

fc=0 | K |=fc 



<^-n{E (au)rei '" (api) ' 



(rf(n + rim)||(a-,t/)|[)' 11 



(bu) Ki ■ ■ ■ (bqi) Ki 



n e 

3=1 k t,>0 



(hj)^ ■ ■ ■ (bqj)^ 
Thus (1) and (2) of the theorem are complete. 



(rf(n + nmmx^y^Y 3 

7T- 1 



-12 
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Notice the fact: (|3.14j) converges absolutely for ||(:r, y)\\ < p implies that the series with y = (0, . . . , 0) 
or x = (0, . . . , 0) also converges absolutely for < p or \\y\\ < p respectively. (3) of the theorem follows 
from divergence of the series associated to Jack polynomials, sec part (3) of Proposition 1, Kaneko □ 
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